README.md
8.6 KB · 192 lines · markdown Raw
1 ---
2 library_name: transformers
3 license: apache-2.0
4 pipeline_tag: image-text-to-text
5 base_model:
6 - Qwen/Qwen3.8-27B
7 tags:
8 - abliterated
9 - uncensored
10 - huihui
11 - qwen3
12 - unsloth
13 - ternary
14 - 2-bit
15 - llama-cpp
16 - cuda
17 - metal
18 - on-device
19 - hybrid-attention
20 - prismml
21 - bonsai
22 ---
23
24 # huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF
25
26
27 This is an uncensored version of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it).
28 This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens.
29
30
31 ## BF16, 1M context
32 **2× RTX 6000 Pro**
33
34 ```
35 llama-server \
36 -m huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-bf16.gguf \
37 --mmproj huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/mmproj-model-bf16.gguf \
38 --port 8080 --host 0.0.0.0 \
39 -ngl 99 \
40 -fa on \
41 -c 1048576 \
42 -np 1 \
43 --rope-scaling yarn \
44 --rope-scale 4 \
45 --yarn-orig-ctx 262144 \
46 --override-kv qwen2.context_length=int:1048576 \
47 --cache-type-k bf16 \
48 --cache-type-v bf16 \
49 -b 4096 \
50 -ub 1024 \
51 --jinja
52
53 ```
54
55
56 ## patch-unsloth 9
57
58 The newly added **Huihui-Qwen3.8-27B-abliterated-patch-all-5.gguf** contains **27** newly added weights and supports the quantized versions of **unsloth/Qwen3.8-27B-GGUF** updated on November 11, 2026.
59
60 The **patch-unsloth** here can patch all quantized variants of the latest version of [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF), from **UD-IQ1_S** to **BF16**, which saves storage space; you need to run the command yourself to apply the patch. For details,
61 Please refer to [README-patch-unsloth.md](patch-unsloth/README-patch-unsloth.md) in the [patch-unsloth](https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/tree/main/patch-unsloth) directory.
62
63
64 ## Latest update 8
65
66 The newly added **Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki** series come from [ConwayResearch/Underdog-Saluki-27B-1.0](https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0) and [ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF).
67 Only layers **22 to 52 (0-based indexing)** have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning.
68
69 Due to ConwayResearch/Underdog-Saluki-27B-1.0 not providing an imatrix file, we will attempt to perform ablation by mixing the GGUF weights of Swift-1.5 and ConwayResearch/Underdog-Saluki-27B-1.0.
70
71 ConwayResearch/Underdog-Saluki-27B-1.0/Underdog-Saluki-27B-1.0-IQ2-mix.gguf only ablated 62 weights.
72
73 This is just a test/validation.
74
75 ## Latest update 7
76
77 The newly added **Huihui-Qwen3.8-27B-abliterated-Swift** series come from [ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF).
78 Only layers **22 to 52 (0-based indexing)** have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning.
79 This is just a test/validation.
80
81 ## Latest update 6
82
83 The newly added **Huihui-Qwen3.8-27B-abliterated-Ternary** series come from [prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
84 ).
85 Only layers **22 to 52 (0-based indexing)** have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning.
86 The size after conversion may differ from the original GGUF (Some of the weights are converted from PTQ1 to Q2_K or Q3_K.).
87 This is just a test/validation.
88
89 The ternary hybrid-attention kernels live in the
90 [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) fork. **Stock llama.cpp will not run these files.**
91
92 ## Latest update 5
93
94 The newly added **Huihui-Qwen3.8-27B-abliterated-GSQ-RCO** series come from [ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
95 ).
96 Only layers **22 to 52 (0-based indexing)** have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning.
97 The size after conversion may differ from the original GGUF.
98
99 ## Latest update 4
100
101 The newly added **Huihui-Qwen3.8-27B-abliterated-UD-DW** series come from [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF).
102 Only layers **22 to 52 (0-based indexing)** have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning.
103 The size after conversion may differ from the original GGUF.
104
105 ## Latest update
106
107 The newly added **Huihui-Qwen3.8-27B-abliterated-UD** series come from [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF).
108 Only layers **17 to 52 (0-based indexing)** have been ablated(Previously, The first 15 layers were retained without ablation), while the other layers remain unablated.
109 The size after conversion may differ from the original GGUF.
110
111 Huihui-Qwen3.8-27B-abliterated-bf16.gguf has also been updated.
112
113 This helps retain more of the original model’s performance. MTP and visual has not been modified.
114
115 ## Note
116 The first 15 layers were retained without ablation. MTP and visual has not been modified.
117
118 We have already converted the weights (token_embd,output,ffn_down,ssm_out,attn_output) that need to be ablated in the versions below Q8_0 from Q2_K, Q3_K, Q4_K, Q5_K, and Q6_K to Q8_0 to improve response quality, and changed the filename to K_L.
119
120 In the Q8_0 quantized version, we changed the Q8_0 weights (token_embd,output,ffn_down,ssm_out,attn_output) targeted for ablation to BF16 and renamed the file to Q8_0_L.
121
122 This is not a standard quantization, so you might find that Q2_K_L is larger than Q3_K and Q4_K.
123
124 ## Specific Quantification Method
125 Some people may misunderstand. The specific quantification method is as follows.
126
127 ### Q2_K_L - Q6_K_L
128
129 [Qwen3.8-27B-tensor_types-Q6_K_L.txt](Qwen3.8-27B-tensor_types-Q6_K_L.txt)
130 ```
131 llama-quantize \
132 --allow-requantize \
133 --tensor-type-file huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Qwen3.8-27B-tensor_types-Q6_K_L.txt \
134 huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-bf16.gguf \
135 huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-Q6_K_L.gguf Q6_K
136 ```
137
138 ### Q8_0_L
139
140 [Qwen3.8-27B-tensor_types-Q8_0_L.txt](Qwen3.8-27B-tensor_types-Q8_0_L.txt)
141
142 ```
143 llama-quantize \
144 --allow-requantize \
145 --tensor-type-file huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Qwen3.8-27B-tensor_types-Q8_0_L.txt \
146 huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-bf16.gguf \
147 huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-Q8_0_L.gguf Q8_0
148
149 ```
150
151
152 ## ollama
153
154 Please use the latest version of [ollama](https://github.com/ollama/ollama/releases)
155
156 You can use [huihui_ai/Qwen3.8-abliterated](https://ollama.com/huihui_ai/Qwen3.8-abliterated) directly,
157 ```
158 ollama run huihui_ai/Qwen3.8-abliterated
159 ```
160
161
162 ## llama.cpp
163 Use the latest [llama.cpp](https://github.com/ggml-org/llama.cpp),
164 ```
165 llama-cli -m huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF/Huihui-Qwen3.8-27B-abliterated-Q4_K.gguf -c 262144
166 ```
167
168 ### Usage Warnings
169
170
171 - **Risk of Sensitive or Controversial Outputs**: This model’s safety filtering has been significantly reduced, potentially generating sensitive, controversial, or inappropriate content. Users should exercise caution and rigorously review generated outputs.
172
173 - **Not Suitable for All Audiences**: Due to limited content filtering, the model’s outputs may be inappropriate for public settings, underage users, or applications requiring high security.
174
175 - **Legal and Ethical Responsibilities**: Users must ensure their usage complies with local laws and ethical standards. Generated content may carry legal or ethical risks, and users are solely responsible for any consequences.
176
177 - **Research and Experimental Use**: It is recommended to use this model for research, testing, or controlled environments, avoiding direct use in production or public-facing commercial applications.
178
179 - **Monitoring and Review Recommendations**: Users are strongly advised to monitor model outputs in real-time and conduct manual reviews when necessary to prevent the dissemination of inappropriate content.
180
181 - **No Default Safety Guarantees**: Unlike standard models, this model has not undergone rigorous safety optimization. huihui.ai bears no responsibility for any consequences arising from its use.
182
183
184 ### Donation
185 ##### Your donation helps us continue our further development and improvement, a cup of coffee can do it.
186 - bitcoin:
187 ```
188 bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
189 ```
190 - Support our work on [Ko-fi](https://ko-fi.com/huihuiai)!
191
192