README.md
12.8 KB · 440 lines · markdown Raw
1 ---
2 tags:
3 - fp8
4 - vllm
5 - llm-compressor
6 - compressed-tensors
7 library_name: transformers
8 license: apache-2.0
9 license_link: https://ai.google.dev/gemma/docs/gemma_4_license
10 pipeline_tag: image-text-to-text
11 base_model: google/gemma-4-31B-it
12 provider: Google
13 name: RedHatAI/gemma-4-31B-it-FP8-block
14 description: FP8-Block variant of gemma-4-31B-it.
15 readme: https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-block/blob/main/README.md
16 tool_calling_supported: true
17 required_cli_args: ['--reasoning-parser gemma4', '--enable-prefix-caching']
18 default-chat-template-kwargs: '{"enable_thinking": true}'
19 chat_template_file_name: None
20 chat_template_path: None
21 tool_call_parser: gemma4
22 validated_tasks:
23 - tool-calling
24 tasks:
25 - text-to-text
26 - text-generation
27 - tool-calling
28 ---
29
30 # gemma-4-31B-it-FP8-block
31
32 ## Model Overview
33 - **Model Architecture:** Gemma4ForConditionalGeneration
34 - **Input:** Text / Image
35 - **Output:** Text
36 - **Model Optimizations:**
37 - **Weight quantization:** FP8
38 - **Activation quantization:** FP8
39 - **Release Date:** 2026-04-04
40 - **Version:** 1.0
41 - **Model Developers:** RedHatAI
42
43 This model is a quantized version of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it).
44 It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
45
46 ### Model Optimizations
47
48 This model was obtained by quantizing the weights and activations of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) to FP8 data type, ready for inference with vLLM.
49 This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
50
51 Weights are quantized using block-wise FP8 scaling (128×128 blocks), and activations are quantized dynamically per group (group_size=128). Only the weights and activations of the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Vision tower, embedding, and output head layers are kept in their original precision.
52
53 ## Deployment
54
55 ### Use with vLLM
56
57 This model can be deployed using [vLLM](https://docs.vllm.ai/en/latest/).
58 For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the [Gemma 4 vLLM usage guide](https://recipes.vllm.ai/Google/gemma-4-31B-it).
59
60 1. Start the vLLM server:
61 ```
62 vllm serve RedHatAI/gemma-4-31B-it-FP8-block \
63 --tensor-parallel-size 2 \
64 --max-model-len 32768 \
65 --gpu-memory-utilization 0.90 \
66 --enable-auto-tool-choice \
67 --reasoning-parser gemma4 \
68 --tool-call-parser gemma4 \
69 --chat-template examples/tool_chat_template_gemma4.jinja \
70 --limit-mm-per-prompt '{"image": 4, "audio": 1}' \
71 --async-scheduling
72 ```
73
74 > **Tip:** For text-only workloads, pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip vision encoder memory allocation and free up GPU memory for a longer context window.
75
76 2. Send requests to the server:
77
78 ```python
79 from openai import OpenAI
80
81 openai_api_key = "EMPTY"
82 openai_api_base = "http://<your-server-host>:8000/v1"
83
84 client = OpenAI(
85 api_key=openai_api_key,
86 base_url=openai_api_base,
87 )
88
89 model = "RedHatAI/gemma-4-31B-it-FP8-block"
90
91 messages = [
92 {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
93 ]
94
95 outputs = client.chat.completions.create(
96 model=model,
97 messages=messages,
98 extra_body={"chat_template_kwargs": {"enable_thinking": True}},
99 )
100
101 generated_text = outputs.choices[0].message.content
102 print(generated_text)
103 ```
104
105 ## Creation
106
107 This model was created by applying data-free FP8 block quantization with [LLM Compressor](https://github.com/vllm-project/llm-compressor), as presented in the code snippet below.
108
109 <details>
110
111 ```python
112 from llmcompressor import model_free_ptq
113
114 MODEL_ID = "google/gemma-4-31B-it"
115 SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-block"
116
117 model_free_ptq(
118 model_stub=MODEL_ID,
119 save_directory=SAVE_DIR,
120 scheme="FP8_BLOCK",
121 ignore=["re:.*vision.*", "lm_head", "re:.*embed_tokens.*"],
122 max_workers=8,
123 device="cuda:0",
124 )
125 ```
126
127 </details>
128
129 ## Evaluation
130
131 This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, AIME 2025, GPQA Diamond, LiveCodeBench v6, and [BFCLv4](https://gorilla.cs.berkeley.edu/leaderboard.html) (function calling) using [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness), [lighteval](https://github.com/neuralmagic/lighteval), and BFCL — all served with [vLLM](https://docs.vllm.ai/en/latest/) (OpenAI-compatible API). Accuracy results are reported both without and with thinking enabled; BFCLv4 was evaluated with thinking enabled. LiveCodeBench v6 was evaluated without thinking only.
132
133 ### Accuracy
134
135 #### Without thinking
136
137 <table>
138 <thead>
139 <tr>
140 <th>Category</th>
141 <th>Benchmark</th>
142 <th>google/gemma-4-31B-it</th>
143 <th>RedHatAI/gemma-4-31B-it-FP8-block</th>
144 <th>Recovery</th>
145 </tr>
146 </thead>
147 <tbody>
148 <tr>
149 <td rowspan="2"><b>Instruction Following</b></td>
150 <td>IFEval (0-shot, prompt-level strict)</td>
151 <td>90.70</td>
152 <td>91.25</td>
153 <td>100.6%</td>
154 </tr>
155 <tr>
156 <td>IFEval (0-shot, inst-level strict)</td>
157 <td>93.45</td>
158 <td>94.00</td>
159 <td>100.6%</td>
160 </tr>
161 <tr>
162 <td rowspan="5"><b>Reasoning</b></td>
163 <td>GSM8K Platinum (0-shot, strict-match)</td>
164 <td>95.78</td>
165 <td>95.78</td>
166 <td>100.0%</td>
167 </tr>
168 <tr>
169 <td>MMLU-Pro (0-shot, custom-extract)</td>
170 <td>85.41</td>
171 <td>85.44</td>
172 <td>100.0%</td>
173 </tr>
174 <tr>
175 <td>MATH-500 (0-shot, pass@1)</td>
176 <td>89.40</td>
177 <td>88.67</td>
178 <td>99.2%</td>
179 </tr>
180 <tr>
181 <td>AIME 2025 (0-shot, pass@1)</td>
182 <td>65.83</td>
183 <td>68.33</td>
184 <td>103.8%</td>
185 </tr>
186 <tr>
187 <td>GPQA Diamond (0-shot, pass@1)</td>
188 <td>77.44</td>
189 <td>77.95</td>
190 <td>100.7%</td>
191 </tr>
192 <tr>
193 <td><b>Coding</b></td>
194 <td>LiveCodeBench v6 (0-shot, pass@1)</td>
195 <td>71.43</td>
196 <td>73.52</td>
197 <td>102.9%</td>
198 </tr>
199 </tbody>
200 </table>
201
202 #### With thinking
203
204 <table>
205 <thead>
206 <tr>
207 <th>Category</th>
208 <th>Benchmark</th>
209 <th>google/gemma-4-31B-it</th>
210 <th>RedHatAI/gemma-4-31B-it-FP8-block</th>
211 <th>Recovery</th>
212 </tr>
213 </thead>
214 <tbody>
215 <tr>
216 <td rowspan="2"><b>Instruction Following</b></td>
217 <td>IFEval (0-shot, prompt-level strict)</td>
218 <td>94.58</td>
219 <td>94.21</td>
220 <td>99.6%</td>
221 </tr>
222 <tr>
223 <td>IFEval (0-shot, inst-level strict)</td>
224 <td>96.44</td>
225 <td>96.12</td>
226 <td>99.7%</td>
227 </tr>
228 <tr>
229 <td rowspan="5"><b>Reasoning</b></td>
230 <td>GSM8K Platinum (0-shot, strict-match)</td>
231 <td>96.11</td>
232 <td>96.20</td>
233 <td>100.1%</td>
234 </tr>
235 <tr>
236 <td>MMLU-Pro (0-shot, custom-extract)</td>
237 <td>87.05</td>
238 <td>87.01</td>
239 <td>100.0%</td>
240 </tr>
241 <tr>
242 <td>MATH-500 (0-shot, pass@1)</td>
243 <td>87.93</td>
244 <td>88.07</td>
245 <td>100.2%</td>
246 </tr>
247 <tr>
248 <td>AIME 2025 (0-shot, pass@1)</td>
249 <td>87.08</td>
250 <td>87.50</td>
251 <td>100.5%</td>
252 </tr>
253 <tr>
254 <td>GPQA Diamond (0-shot, pass@1)</td>
255 <td>86.36</td>
256 <td>86.87</td>
257 <td>100.6%</td>
258 </tr>
259 <tr>
260 <td rowspan="4"><b>Tool Calling</b></td>
261 <td>BFCLv4 Overall</td>
262 <td>70.77%</td>
263 <td>67.59%</td>
264 <td>95.5%</td>
265 </tr>
266 <tr>
267 <td>BFCLv4 Single Turn</td>
268 <td>84.99%</td>
269 <td>85.15%</td>
270 <td>100.2%</td>
271 </tr>
272 <tr>
273 <td>BFCLv4 Multi-Turn</td>
274 <td>66.25%</td>
275 <td>67.62%</td>
276 <td>102.1%</td>
277 </tr>
278 <tr>
279 <td>BFCLv4 Agentic</td>
280 <td>65.82%</td>
281 <td>56.53%</td>
282 <td>85.9%</td>
283 </tr>
284 </tbody>
285 </table>
286
287 ### Reproduction
288
289 The results were obtained using the following commands:
290
291 <details>
292
293 Each benchmark was run 3 times with different random seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds.
294
295 **vLLM server (with thinking):**
296 ```
297 vllm serve RedHatAI/gemma-4-31B-it-FP8-block \
298 --served-model-name gemma-4-31b-it-FP8-block \
299 --max-model-len 32768 \
300 --gpu-memory-utilization 0.90 \
301 --language-model-only \
302 --enable-auto-tool-choice \
303 --reasoning-parser gemma4 \
304 --tool-call-parser gemma4 \
305 --chat-template examples/tool_chat_template_gemma4.jinja \
306 --async-scheduling \
307 --default-chat-template-kwargs '{"enable_thinking": true}'
308 ```
309
310 > **Note:** To reproduce the results without thinking, remove `--default-chat-template-kwargs '{"enable_thinking": true}'`. To run without tool calling, remove `--enable-auto-tool-choice`, `--tool-call-parser gemma4`, and `--reasoning-parser gemma4`.
311
312 #### GSM8K Platinum (lm-eval, 0-shot, 3 repetitions)
313 ```
314 lm_eval --model local-chat-completions \
315 --tasks gsm8k_platinum_cot_llama \
316 --model_args "model=gemma-4-31b-it-FP8-block,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
317 --num_fewshot 0 \
318 --apply_chat_template \
319 --output_path results_gsm8k_platinum.json \
320 --seed 1234 \
321 --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
322 ```
323
324 #### MMLU-Pro (lm-eval, 0-shot, 3 repetitions)
325 ```
326 lm_eval --model local-chat-completions \
327 --tasks mmlu_pro_chat \
328 --model_args "model=gemma-4-31b-it-FP8-block,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
329 --num_fewshot 0 \
330 --apply_chat_template \
331 --output_path results_mmlu_pro.json \
332 --seed 1234 \
333 --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
334 ```
335
336 #### IFEval (lm-eval, 0-shot, 3 repetitions)
337 ```
338 lm_eval --model local-chat-completions \
339 --tasks ifeval \
340 --model_args "model=gemma-4-31b-it-FP8-block,max_length=32768,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
341 --num_fewshot 0 \
342 --apply_chat_template \
343 --output_path results_ifeval.json \
344 --seed 1234 \
345 --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
346 ```
347
348 #### MATH-500, AIME 2025, GPQA Diamond (lighteval, 3 repetitions; 8 for AIME 2025)
349
350 **litellm_config.yaml:**
351 ```yaml
352 model_parameters:
353 provider: hosted_vllm
354 model_name: hosted_vllm/gemma-4-31b-it-FP8-block
355 base_url: http://0.0.0.0:8000/v1
356 api_key: ''
357 timeout: 3600
358 concurrent_requests: 32
359 generation_parameters:
360 temperature: 1.0
361 max_new_tokens: 65536
362 top_p: 0.95
363 top_k: 64
364 seed: 1234
365 ```
366
367 Run once per seed (changing `seed` in the config each time):
368 ```
369 lighteval endpoint litellm litellm_config.yaml 'math_500|0' \
370 --output-dir results/ --save-details
371
372 lighteval endpoint litellm litellm_config.yaml 'aime25|0' \
373 --output-dir results/ --save-details
374
375 lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \
376 --output-dir results/ --save-details
377 ```
378
379 #### LiveCodeBench v6 (lighteval, 3 repetitions, without thinking)
380
381 **litellm_config.yaml:**
382 ```yaml
383 model_parameters:
384 provider: hosted_vllm
385 model_name: hosted_vllm/gemma-4-31b-it-FP8-block
386 base_url: http://0.0.0.0:8000/v1
387 api_key: ''
388 timeout: 1200
389 concurrent_requests: 32
390 generation_parameters:
391 temperature: 1.0
392 max_new_tokens: 32768
393 top_p: 0.95
394 top_k: 64
395 seed: 1234
396 ```
397
398 Run once per seed (using the vLLM server without `--default-chat-template-kwargs`):
399 ```
400 lighteval endpoint litellm litellm_config.yaml 'lcb:codegeneration_v6|0' \
401 --output-dir results/ --save-details
402 ```
403
404 #### BFCLv4
405
406 BFCL requires the model to be registered in the leaderboard codebase before running evaluation.
407
408 **Step 1 — Register the model in `bfcl_eval/constants/model_config.py`**
409
410 Add the following entry to `api_inference_model_map`:
411
412 ```python
413 "gemma-4-31b-it-FP8-block": ModelConfig(
414 model_name="gemma-4-31b-it-FP8-block",
415 display_name="Gemma-4-31b-it-FP8-Block (FC)",
416 url="https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-block",
417 org="Google",
418 license="Apache 2.0",
419 model_handler=OpenAICompletionsHandler,
420 input_price=None,
421 output_price=None,
422 is_fc_model=True,
423 underscore_to_dot=True,
424 ),
425 ```
426
427 **Step 2 — Add the key to `bfcl_eval/constants/supported_models.py`**
428
429 Add `"gemma-4-31b-it-FP8-block"` to the `SUPPORTED_MODELS` list.
430
431 **Step 3 — Start the vLLM server** (use the command at the top of this section; the `--served-model-name` flag ensures BFCL can find the model by its registered slug).
432
433 **Step 4 — Generate responses and evaluate**
434 ```
435 bfcl generate --model gemma-4-31b-it-FP8-block --test-category all
436 bfcl evaluate --model gemma-4-31b-it-FP8-block --test-category all
437 ```
438
439 </details>
440