KNOWN_ISSUES.md
17.5 KB · 194 lines · markdown Raw
1 # Ternary Bonsai 2 27B: known issues
2
3 Last checked: 2026-09-23. Applies to `Ternary-Bonsai-2-27B` (GGUF `PQ2_0` / `PTQ1_0` / `Q2_0` / `F16`, and the MLX build).
4
5 This page lists problems we know about, with the fastest workaround first. If you hit something that is not here, please open an issue on [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp/issues) (runtime) or [PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo/issues) (setup scripts and demo server).
6
7 **Status labels**
8
9 - **Fixed in release**: available in a published binary release.
10 - **Fixed in source**: merged to the `prism` branch of PrismML-Eng/llama.cpp. It isn't in a binary release yet, so build from source or wait for the next release.
11 - **Fix in review**: a pull request exists and has not been merged.
12 - **Open**: no fix yet.
13
14 ## Start here: three fixes that cover most reports
15
16 | If you see... | Do this |
17 |---|---|
18 | Empty answers, or "it thinks forever and never answers" | Give the model a large output limit: `-n 16384` or more (`max_tokens` for the API), with `-c 65536`. For shorter thinking, also send `reasoning_effort: "medium"`. |
19 | HTTP 500 with `reasoning_effort: "high"` | Use `"xhigh"` (the default) or `"medium"`. |
20 | Crash on load on an AMD Zen 4/Zen 5 or other AVX-512 CPU | See [CPU crash on load](#cpu-crash-on-load-avx-512-cpus). |
21
22 ## Empty, cut-off or runaway output
23
24 **Empty or truncated answers.** The model reasons before it answers, and the reasoning counts against the output limit. If the limit runs out while it's still reasoning, you get an empty or cut-off reply. This is the most common report we see, including reports of poor output in non-English languages.
25 *Workaround:* `-n 16384` or higher (or `max_tokens`), with `-c 65536`. A small cap such as `-n 256` ends generation mid-thought. `reasoning_effort: "medium"` also helps: it runs out of budget far less often (see [Long reasoning](#empty-cut-off-or-runaway-output)). **Status: documented in the model card.**
26
27 **`reasoning_effort: "high"` returns HTTP 500.** The chat template accepts `low`, `medium` and `xhigh`, not `high`.
28 *Workaround:* use `xhigh` (the default) or `medium`. **Status: open.**
29
30 **`low` doesn't shorten reasoning much.** Although the chat template asks for brief reasoning at `low`, in our measurements it used about as many tokens as the default `xhigh`.
31 *Workaround:* use `medium` for shorter responses, as the model card recommends, or set a thinking budget. **Status: known behaviour.**
32
33 **`low` can still run out of output budget.** Under a tight output limit, `low` doesn't always finish.
34 *Workaround:* keep the output limit generous (see above). **Status: open.**
35
36 **A thinking budget in `chat_template_kwargs` is ignored.** The chat template has no thinking-budget hook, so `chat_template_kwargs.thinking_budget_tokens` has no effect and produces no error.
37 *Workaround:* send the budget as a top-level field of the request body, or start the server with `--reasoning-budget N`; a value in the request overrides the server flag. A budget of `0` does not turn reasoning off. To turn it off, send `reasoning_effort: "none"` with the server left at `--reasoning auto` (the default). **Status: open (documentation).**
38
39 **`--reasoning on` overrides a client's `reasoning_effort: "none"`.**
40 *Workaround:* leave the server at `--reasoning auto`. **Status: open.**
41
42 **Looping on some prompts at `xhigh`.** A small share of instruction-following prompts loop instead of finishing. Community reports suggest `--repeat-penalty 1.05` helps. In our tests it didn't help noticeably, and a stronger penalty tended to hurt.
43 *Workaround:* `reasoning_effort: "medium"`, which runs out of budget much less often. **Status: open.**
44
45 **Long reasoning.** The model tends to reason at length, which costs time and tokens even when the answer is correct.
46 *Workaround:* use `reasoning_effort: "medium"` or a thinking budget. When the output limit is moderate (around 16k tokens), `medium` thinks noticeably less than the default `xhigh` and is about as accurate, because `xhigh` more often runs out of budget mid-reasoning. With much larger output budgets, `xhigh` is the more accurate setting. **Status: known behaviour.**
47
48 ## Sampling and metadata
49
50 **`min_p` is not in the GGUF.** The file carries `top_k=20`, `top_p=0.95` and `temperature=1.0`, but not `min_p`. The recommended value is 0.05, which is llama.cpp's own default, so llama.cpp users get it automatically.
51 *Workaround:* with a runtime whose default is 0.0, set `min_p` to 0.05. **Status: planned** (a metadata update to the model files will add the key).
52
53 **`presence_penalty` and `repetition_penalty` are not in the GGUF**, so a server started without options won't apply the model card's values.
54 *Workaround:* set them explicitly as the model card describes. **Status: open.**
55
56 **MLX build samples almost greedily.** The MLX `generation_config.json` is missing its sampling settings.
57 *Workaround:* pass the sampler settings explicitly (`temperature=1.0`, `top_p=0.95`, `top_k=20`). **Status: open.**
58
59 **The website and the model card list different sampling settings.** The model card is correct. **Status: open (documentation).**
60
61 **The GGUF's name field reads "Hf".** This is cosmetic, but tools that display or catalogue model metadata show the wrong name. **Status: fix in review** (a metadata update to the model files sets the name).
62
63 ## Tool calling and agent loops
64
65 **Malformed tool calls.** Tool calls sometimes come out malformed or loop (for example repeated `// // //`). This is a known model limitation in structured output, and we're working on it. **Status: open.**
66
67 **HTTP 500/400 when a tool call has empty or non-JSON `arguments`.**
68 *Workaround:* send `"{}"` for a call with no arguments. **Status: open** ([#112](https://github.com/PrismML-Eng/llama.cpp/issues/112)).
69
70 **HTTP 500 when a system message isn't first, or when there are two.** The template requires exactly one system message, at the start.
71 *Workaround:* merge system messages into one leading message. **Status: open.**
72
73 **Context grows and the prompt cache misses in tool loops.** Reasoning from earlier turns is re-rendered into later ones, and tool calls are re-rendered differently from how they were generated. Each turn then re-processes most of the conversation.
74 *Workaround:* don't echo reasoning back into the conversation in tool loops. **Status: open** ([Bonsai-demo #183](https://github.com/PrismML-Eng/Bonsai-demo/issues/183)).
75
76 ## Crashes and load failures
77
78 ### CPU crash on load (AVX-512 CPUs)
79
80 **`PQ2_0` segfaults while loading on CPUs with AVX-512**, including AMD Zen 4 and Zen 5 (Ryzen AI 300 series, Strix Halo) and some server and virtualised CPUs. It crashes even with all layers offloaded to a GPU.
81 *Workaround:* build from source at the current `prism` branch. `--no-repack` may avoid it but is unconfirmed. **Status: fixed in source** ([#245](https://github.com/PrismML-Eng/llama.cpp/pull/245), merged 2026-09-23; reports [#180](https://github.com/PrismML-Eng/llama.cpp/issues/180), [#204](https://github.com/PrismML-Eng/llama.cpp/issues/204), [#219](https://github.com/PrismML-Eng/llama.cpp/issues/219), [Bonsai-demo #182](https://github.com/PrismML-Eng/Bonsai-demo/issues/182)).
82
83 **CUDA 13.3 builds crash on some systems.** On Linux this is a segfault; on Windows the server prints its banner and exits without a message.
84 *Workaround:* use the CUDA 12.8 build on Linux or the CUDA 12.4 build on Windows. **Status: open** ([#222](https://github.com/PrismML-Eng/llama.cpp/issues/222)).
85
86 **Windows CUDA builds don't start on some CPUs**; the Windows CPU-only build does.
87 *Workaround:* use the CPU-only build while this is investigated. **Status: open** ([#241](https://github.com/PrismML-Eng/llama.cpp/issues/241)).
88
89 **ROCm/HIP aborts on consumer RDNA2 GPUs** (for example gfx1030).
90 *Workaround:* none yet on HIP; try the Vulkan build. **Status: open** ([Bonsai-demo #197](https://github.com/PrismML-Eng/Bonsai-demo/issues/197)).
91
92 **`PQ2_0` produces garbled output on Windows HIP with gfx1151** (Strix Halo integrated GPU). CPU (`-ngl 0`) output is correct.
93 *Workaround:* `-ngl 0`, or the Vulkan build with `PTQ1_0`. **Status: open** ([#223](https://github.com/PrismML-Eng/llama.cpp/issues/223)).
94
95 **`--split-mode tensor` and `--split-mode row` can't load across two GPUs.**
96 *Workaround:* use the default layer split. **Status: open** ([#213](https://github.com/PrismML-Eng/llama.cpp/issues/213)).
97
98 **`Q2_0` files from older Bonsai repositories won't load.** They declare type 42 but store a 128-weight block layout, and current builds, stock and PrismML alike, read type 42 as the 64-weight layout.
99 *Workaround:* use the `Q2_0` file in this repository, which uses the current layout. **Status: answered** ([#167](https://github.com/PrismML-Eng/llama.cpp/issues/167)).
100
101 ### Runtimes other than the PrismML build
102
103 `PQ2_0` and `PTQ1_0` are new quantization types, and support for them isn't in mainline llama.cpp yet. That means:
104
105 - Stock llama.cpp, Ollama and LM Studio (GGUF) can't load the `PQ2_0` or `PTQ1_0` files.
106 - The `F16` file runs correctly on stock llama.cpp from b11556 onward, like `Q2_0`. Older stock builds load it without any warning and produce garbled output, because they don't apply the Hadamard metadata it depends on.
107 - The `Q2_0` file runs correctly on stock llama.cpp from b11556 onward. Older stock builds, and apps that bundle one, load it without any warning and produce garbled output. Use b11556 or newer.
108
109 *Workaround:* use the [PrismML llama.cpp build](https://github.com/PrismML-Eng/llama.cpp/releases). **Status: open.**
110
111 **The MLX build needs its bundled loader.** MLX apps and LM Studio (MLX) can't load it yet. Support has landed upstream in mlx-vlm but isn't in a released runtime.
112 *Workaround:* load with the loader shipped beside the model. Also use mlx-vlm 0.7.1 or newer: 0.6.3 breaks image input when serving. **Status: open.**
113
114 **Quantizing your own model to `PQ2_0` or `PTQ1_0` produces garbage.** `llama-quantize` offers both types, but they only work on checkpoints exported with Prism's Hadamard rotation, which also writes the `prism.hadamard.*` metadata. An ordinary model quantized to either type has no rotation metadata, so it loads without any warning and generates nonsense.
115 *Workaround:* use the released Bonsai files; for your own models, use the standard quantization types. **Status: open** ([#242](https://github.com/PrismML-Eng/llama.cpp/issues/242)).
116
117 **The MLX build of the 27B model has no vision support.**
118 *Workaround:* for image input, use the GGUF with the `mmproj` file. **Status: open.**
119
120 ## Downloads, setup and building from source
121
122 **The launch-day release (`prism-b10687`) has no binaries.**
123 *Workaround:* use [`prism-b10709`](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10709-9a9394a) or newer, which has builds for macOS, Linux, Windows and Android. Please prefer official releases to copies shared elsewhere. **Status: fixed in release b10709** ([#183](https://github.com/PrismML-Eng/llama.cpp/issues/183), [#193](https://github.com/PrismML-Eng/llama.cpp/issues/193), [#195](https://github.com/PrismML-Eng/llama.cpp/issues/195)).
124
125 **Windows `setup.ps1` installed the Vulkan build on NVIDIA machines**, which runs far slower than CUDA and shows no error.
126 *Workaround:* update the demo repository, then check that the server log reports a CUDA device. **Status: fixed** ([Bonsai-demo #176](https://github.com/PrismML-Eng/Bonsai-demo/issues/176)).
127
128 **Windows `setup.ps1` downloads only `PQ2_0`.**
129 *Workaround:* download `PTQ1_0` manually from the model repository if you need it (for example for Vulkan). **Status: open** ([Bonsai-demo #184](https://github.com/PrismML-Eng/Bonsai-demo/issues/184)).
130
131 **`setup.sh` downloads about 19 GB of model files without asking.**
132 *Workaround:* set `BONSAI_SKIP_GGUF=1` to skip the download. **Status: open (documentation).**
133
134 **The default 32K context doesn't fit on 12 GB GPUs.** The demo scripts size context from system RAM, not GPU memory.
135 *Workaround:* set `BONSAI_CTX` to a smaller value, or `BONSAI_KV4=1`. **Status: open** ([Bonsai-demo #193](https://github.com/PrismML-Eng/Bonsai-demo/issues/193)).
136
137 **`setup.sh` fails to build on macOS 27 / Xcode 26.5.**
138 *Workaround:* see the patch in the issue. **Status: open** ([Bonsai-demo #177](https://github.com/PrismML-Eng/Bonsai-demo/issues/177)).
139
140 **CUDA builds from source fail with "CUDA_ARCHITECTURES is empty".**
141 *Workaround:* pass `-DCMAKE_CUDA_ARCHITECTURES=<your arch>` (for example `89` for RTX 40-series). **Status: open** ([#182](https://github.com/PrismML-Eng/llama.cpp/issues/182)).
142
143 **CUDA builds may need `-DGGML_CUDA_FA_ALL_QUANTS=ON`** for the quantized KV-cache types to work with flash attention. The build docs don't mention it yet. **Status: open (documentation).**
144
145 **RTX 5090: prebuilt Linux binaries decode erratically.**
146 *Workaround:* build from source with `-DCMAKE_CUDA_ARCHITECTURES=120a-real`. **Status: open** ([#199](https://github.com/PrismML-Eng/llama.cpp/issues/199)).
147
148 ## Performance
149
150 **`q5_0` KV cache is several times slower** than the other cache types at all context depths.
151 *Workaround:* use `f16`, `q8_0` or `q4_0` for `-ctk`/`-ctv`. **Status: open** ([#191](https://github.com/PrismML-Eng/llama.cpp/issues/191)).
152
153 **A single conversation keeps re-processing its prompt on the server.** By default the server splits its cache across several slots, and the prompt cache is too small for long conversations.
154 *Workaround:* for single-user use, start the server with `-np 1 --cache-ram 24576`. **Status: open** ([Bonsai-demo #183](https://github.com/PrismML-Eng/Bonsai-demo/issues/183)).
155
156 **`PQ2_0` on Vulkan silently runs on the CPU** in the current binary release, at under 2 tokens per second. The log still reports all layers offloaded to the GPU; the tell is a `CPU model buffer size` of several GB, with almost nothing on the Vulkan device.
157 *Workaround:* build from source at the current `prism` branch, where `PQ2_0` runs fully on the GPU, or use `PTQ1_0`. **Status: fixed in source** ([#238](https://github.com/PrismML-Eng/llama.cpp/pull/238), merged 2026-09-21; not yet in a binary release).
158
159 **`Q2_0` on stock llama.cpp with Vulkan doesn't use the fast Hadamard kernel yet.** Mainline's Vulkan kernel covers blocks up to 512 wide and this model uses 1024, so that step falls back to a generic matrix multiply on the GPU and runs slower than it will once the kernel lands. **Status: fix in review** ([llama.cpp#29772](https://github.com/ggml-org/llama.cpp/pull/29772)).
160
161 **`PTQ1_0` decode on Vulkan is slower than it should be** in the current binary release: it runs on the GPU, but decodes at under 2 tokens per second on an Intel Arc B390. **Status: fixed in source for Intel GPUs that llama.cpp's Vulkan backend classes as Xe2** ([#238](https://github.com/PrismML-Eng/llama.cpp/pull/238), merged 2026-09-21; not yet in a binary release). That covers Intel GPUs that report a 16-wide subgroup, which includes Xe3 parts such as the Panther Lake Arc B390, where it measured about 9x faster. Intel iGPUs that report an 8-wide subgroup, such as the Arc 140T, don't get it yet; a fix is proposed in [#270](https://github.com/PrismML-Eng/llama.cpp/issues/270). Other GPUs are unchanged so far; **further speedups in review** ([#252](https://github.com/PrismML-Eng/llama.cpp/pull/252)). Reports: [#186](https://github.com/PrismML-Eng/llama.cpp/issues/186), [#201](https://github.com/PrismML-Eng/llama.cpp/issues/201).
162
163 **`PTQ1_0` on Vulkan can hang Intel Arc GPUs** after about 1,900 tokens.
164 *Workaround:* run on CPU for now. **Status: open** ([#192](https://github.com/PrismML-Eng/llama.cpp/issues/192)).
165
166 **The SYCL build can't run `PQ2_0` or `PTQ1_0` yet**; it stops at load with "unsupport data type".
167 *Workaround:* on Intel GPUs, use the Vulkan build for now. **Status: fix in review** ([#235](https://github.com/PrismML-Eng/llama.cpp/pull/235)).
168
169 **CPU inference on consumer x86 CPUs** (AMD Zen 1–4, Intel Alder Lake and newer) doesn't use the fast `PQ2_0` kernels in current releases.
170 *Workaround:* build from source at the current `prism` branch for the AVX2/AVX-VNNI kernels. **Status: fixed in source for `PQ2_0`** ([#206](https://github.com/PrismML-Eng/llama.cpp/pull/206), merged 2026-09-21); **fix in review for `PTQ1_0`** ([#250](https://github.com/PrismML-Eng/llama.cpp/pull/250)).
171
172 **The Metal tensor API is not enabled on the newest Apple chips.** **Status: fixed in source** ([#196](https://github.com/PrismML-Eng/llama.cpp/pull/196), merged 2026-09-21).
173
174 **Multi-token prediction (MTP) is refused for these files.** **Status: fixed in source** ([#205](https://github.com/PrismML-Eng/llama.cpp/pull/205), merged 2026-09-21).
175
176 **`--spec-type ngram-*` has no effect on this model.** The flag is accepted but no speculation runs, so throughput reported with it is plain decoding. **Status: open** ([#203](https://github.com/PrismML-Eng/llama.cpp/issues/203)).
177
178 **`PTQ1_0` decode on CUDA.** A faster small-batch kernel is in review. **Status: fix in review** ([#218](https://github.com/PrismML-Eng/llama.cpp/pull/218)).
179
180 **AMD GPUs on HIP decode slower than expected.** **Status: fix in review** ([#116](https://github.com/PrismML-Eng/llama.cpp/pull/116)).
181
182 ## Vision
183
184 **Poor results on "point at the object" (grounding) tasks with small images.**
185 *Workaround:* start the server with `--image-min-tokens 1024`. **Status: open (the demo server doesn't pass it yet).**
186
187 ## Reported, under investigation
188
189 We haven't reproduced these yet. If you can, a report with your command line, hardware and build helps.
190
191 - Stalls between turns after tool calls on AMD ROCm (RX 9070 XT).
192 - Connection resets or crashes with a quantized KV cache at very long contexts (around 200K tokens).
193 - `download_binaries.sh` choosing the CUDA 12.8 build on CUDA 13.x systems ([Bonsai-demo #206](https://github.com/PrismML-Eng/Bonsai-demo/issues/206)).
194