server : realtime: send an error event instead of crashing on exceptions (#31632) server : realtime: send an error event instead of crashing on exceptions server : route websocket handlers through ex_wrapper, remove the realtime test server : realtime: keep the previous tools when the chat template rejects new ones server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapp…
opencl: improve decode perf with FA (#31633) opencl: use more workgroups for q4_K gate up + GLU Co-authored-by: Shaofei Qi shaoqi@qti.qualcomm.com opencl: handle FA for GQA 2 and 6 with head size 256 Co-authored-by: Shaofei Qi shaoqi@qti.qualcomm.com opencl: handle FA for GQA 16 with head size of 128, 512 Co-authored-by: Shaofei Qi shaoqi@qti.qualcomm.com opencl: handle FA for GQA 2 with head size…
model, mtmd: drama-3 tts support (#31613) model, mtmd: drama-3 tts support clean up mtmd: drama3 helper stops at the codebook tower limit Co-authored-by: Pascal admin@serveurperso.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63780976 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS…
webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (#31348) Fix get_max_size and improve the memset shader Fix buffer_memset to handle the buffer size more than max binding size. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63772875 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI…
Signed-off-by: Adrien Gallouët angt@huggingface.co
Overview This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS…
llama.cpp : bump version to 0.9.0 (#31646) llama.cpp : bump version to 0.9.0 authors : update Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63573996 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s3…
sync : ggml Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63566813 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x (zDNN) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubun…
ggml: bound views by the bytes their strides reach (#31584) ggml: bound views by the bytes their strides reach A view was checked against its source with the size of a contiguous tensor of the same shape, computed before its strides were set. Views whose rows overlap failed that check although they stay inside their source, and views with wider strides passed it while reaching past the end. The vi…
ggml : write host-buffer tensor data directly in gguf writer (#31144) The gguf writer staged each tensor into a temporary vector before writing it. For host-buffer tensors the data pointer is directly writable, so skip the staging copy and write it straight to the file. This avoids the per-tensor allocation, zero-init and memcpy. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGU…
opencl: improve q2_0 decoding and small f32 matmuls Assisted-by: Codex opencl: optimize small f32 matmuls on Adreno 8xx Assisted-by: Codex
common : fix -hf tag matching inside longer quant names (#31620) common : fix -hf tag matching inside longer quant names ":F16" picked a BF16 file when the repo had both, since "BF16" contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag now has to start at a token boundary in the path. Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) common : accept…
hexagon: add LIGHTNING_INDEXER support (#31289) hexagon: initial support for LIGHTNING_INDEXER op hex-lght-idxr: hmx for prefill hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute hex-lght-idxr: pipelined hmx path for decode and prefill hex-lid: add support for K (f32) hex-lid: add support for K (f32) hex-lid: add support for q4_0 hex-lid: remove scalar VTCM reads in the HMX q4_…
mtmd: accurate support for qwen3-vl video prompt format (#31619) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63416358 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x…
model, mtmd: add chatterbox TTS support (#30993) add chatterbox TTS support Co-authored-by: Pascal admin@serveurperso.com add built-in voice support rm redundant code clean up clean up 2 Apply batched suggestions from code review Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co improve Co-authored-by: Pascal admin@serveurperso.com Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret…
hexagon: add TOP_K_MASK op (#31552) hexagon: add TOP_K_MASK op Simplify Hexagon TOP_K_MASK kernel and do not duplicate host configuration Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63407530 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu ar…
hexagon: faster TOP_K for the backend sampler (#31279) hexagon: faster TOP_K for the backend sampler (skip tiles, then block scan) Skip the bitonic sort for tiles whose max cannot reach the running k-th best value, then replace the tile sort with top_k_scan(): reject 256-float blocks with one HVX max-reduce and insert only surviving elements into a sorted k-entry list. Used for k in 2..64 and rows…
spec : save and restore MTP hidden state (#30141) speculative : save and restore MTP hidden state Assisted-by: Codex tests : remove MTP state test Assisted-by: Codex speculative : simplify MTP checkpoint state Save only the boundary hidden row; verification rows are rebuilt on the supported restore paths. Assisted-by: GPT-6 Co-authored-by: zouyongxing zouyongxing@bytedance.com Website: https://lla…
ggml-zdnn: fix fattn test failure (#31583) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63384051 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x (zDNN) Ubuntu x64 (Vul…
model : Add support for Metis-1.6 models (#31134) Assisted-by: Claude Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63378637 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s…
model : enable NVFP4 LFM3 ggufs (#31608) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63372395 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x (zDNN) Ubuntu x64 (Vulka…
mtmd : add Nemotron 3.5 Transcribe support (#31471) mtmd : add Nemotron 3.5 Transcribe support Assisted-by: Claude convert : fix vocab of draft models for Nemotron-H targets Assisted-by: Claude convert : reject non-BF16 Transcribe checkpoints Assisted-by: Claude mtmd : make speaker FIFO predictions local Assisted-by: Claude mtmd : normalize the mel in encode_windows Assisted-by: Claude convert : m…
qwen4exp: write the conv state tail without an intermediate cont (#31443) build_conv_state_at makes a contiguous copy of the conv input tail before writing it into the recurrent state, so every GDN layer issues two device copies per generated token where one is enough. On Qwen3.8-Flash-Next (36 GDN layers) that is 72 copies per token, now 36. Speculative decoding adds one more per rollback slot. g…
model : record layer inputs in LFM3 for spec dec (#31598) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63354471 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x (zDNN)…
ggml: avoid signed overflow in ggml_roll bounds check (#30375) ggml_roll validated each shift with abs(shift), but abs(INT_MIN) is signed integer overflow UB evaluated before the bounds check runs. Compare with a signed-safe test instead. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63348301 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (…
cuda: potential fix for Q8_1 correction sum (#31247) cuda: potential fix for Q8_1 correction sum Update ggml/src/ggml-cuda/quantize.cu Co-authored-by: Gaurav Garg gaugarg@nvidia.com addressed review comment to change the quantize_mmq_q8_1 flow as well Co-authored-by: Gaurav Garg gaugarg@nvidia.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63333029…
cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391) cuda: gather batched tensor reads into mapped host memory Assisted-by: Codex add comments Assisted-by: Codex simplify the implementation based on checks that exist earlier in the flow simplify batched tensor gather indexing add conditional dispatch address comments - make host registration raii, better test mindful…
Add nccl support for multi-node tensor parallelism (#28967) Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available. Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators reuse ggml_backend_cuda_comm_allreduce_nccl for per-ra…
gemma4: graph optimizations (massive gains on pp for E2B and E4B) (#3…
tools : reject mismatched shards in GGUF split merge (#31579) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/63239877 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu 26.04 s390x (CPU) Ubuntu 26.04 s390x (zD…