What's Changed misc(gha): expose action cache url and runtime as secrets by @mfuntowicz in #2964 feat: support max_image_fetch_size to limit by @drbh in #3339 Maintenance mode by @LysandreJik in #3344 Maintenance mode by @LysandreJik in #3345 fix(num_devices): fix num_shard/num device auto compute when NVIDIA_VISIBLE_DEVICES == "all" or "void" by @oOraph in #3346 Full Changelog: v3.3.6...v3.3.7
What's Changed Add missing backslash by @philsupertramp in #3311 Revert "feat: bump flake including transformers and huggingface_hub versions" by @drbh in #3323 fix: remove azure by @drbh in #3325 Fix mask passed to flashinfer by @danieldk in #3324 Update iframe sources for streaming demo by @coyotte508 in #3327 Revert "Revert "feat: bump flake including transformers and huggingfa… by @drbh in #33…
What's Changed [gaudi] Refine rope memory, do not need to keep sin/cos cache per layer by @sywangyi in #3274 Gaudi: add CI by @baptistecolle in #3160 [gaudi] Gemma3 sliding window support by @sywangyi in #3280 xpu lora support by @sywangyi in #3232 Optimum neuron 0.2.2 by @dacorvo in #3281 [gaudi] Remove unnecessary reinitialize to HeterogeneousNextTokenChooser to m… by @sywangyi in #3284 [gaudi]…
Fix for Neuron models exported with batch_size 1. What's Changed [gaudi] gemma3 text and vlm model intial support. need to add sliding window … by @sywangyi in #3270 Neuron backend fix by @dacorvo in #3273 Full Changelog: v3.3.3...v3.3.4
Neuron backend update. What's Changed Remove useless packages by @yuanwu2017 in #3253 Bump neuron SDK version by @dacorvo in #3260 Perf opt by @sywangyi in #3256 [gaudi] Vlm rebase and issue fix in benchmark test by @sywangyi in #3263 Move the _update_cos_sin_cache into get_cos_sin by @yuanwu2017 in #3254 [Gaudi] Remove optimum-habana by @yuanwu2017 in #3261 [gaudi] HuggingFaceM4/idefics2-8b issue…
Gaudi improvements. What's Changed upgrade to new vllm extension ops(fix issue in exponential bucketing) by @sywangyi in #3239 Nix: switch to hf-nix by @danieldk in #3240 Add Qwen3 by @yuanwu2017 in #3229 fp8 compressed_tensors w8a8 support by @sywangyi in #3242 [Gaudi] Fix the OOM issue of Llama-4-Scout-17B-16E-Instruct by @yuanwu2017 in #3245 Fix the Llama-4-Maverick-17B-128E crash issue by @yua…
This release updates TGI to Torch 2.7 and CUDA 12.8. What's Changed change HPU warmup logic: seq length should be with exponential growth by @kaixuanliu in #3217 adjust the round_up_seq logic to align with prefill warmup phase on… by @kaixuanliu in #3224 Update to Torch 2.7.0 by @danieldk in #3221 Enable Llama4 for gaudi backend by @yuanwu2017 in #3223 fix: count gpu uuids if NVIDIA_VISIBLE_DEVICE…
Notable changes Prefill chunking for VLMs. What's Changed Fixing Qwen 2.5 VL (32B). by @Narsil in #3157 Fixing tokenization like https://github.com/huggingface/text-embeddin… by @Narsil in #3156 Gaudi: clean cuda/rocm code in hpu backend, enable flat_hpu by @sywangyi in #3113 L4 fixes by @mht-sharma in #3161 setuptools <= 70.0 is vulnerable: CVE-2024-6345 by @Narsil in #3171 transformers flash llm…
Main changes Patching Llama 4 What's Changed Use ROCM 6.3.1 by @mht-sharma in #3141 Update transformers to 4.51 by @mht-sharma in #3148 Gaudi: Add Integration Test for Gaudi Backend by @baptistecolle in #3142 fix: compute type typo by @oOraph in #3150 3.2.3 by @Narsil in #3151 Full Changelog: v3.2.2...v3.2.3
What's Changed Minor fixes. by @Narsil in #3125 configurable termination timeout by @ErikKaum in #3126 CI: enable server tests for backends by @baptistecolle in #3128 Torch 2.6 by @Narsil in #3134 Gaudi: Fix llava-next and mllama crash issue by @yuanwu2017 in #3127 nix-v3.2.1 -> v3.2.1-nix by @co42 in #3129 Gaudi: Use exponential growth to replace BATCH_BUCKET_SIZE by @yuanwu2017 in #3131 Add llam…