README.md
16.6 KB · 385 lines · markdown Raw
1 ---
2 language:
3 - en
4 - zh
5 - ja
6 - ko
7 - fr
8 - es
9 - pt
10 - de
11 - it
12 - ru
13 - ar
14 - vi
15 - th
16 tags:
17 - text-generation
18 - deltanet
19 - qwen3.8
20 - gguf
21 - cpu
22 - efficient
23 - low-memory
24 - ternary
25 - bitnet
26 - jirack
27 - web-ui
28 - routing
29 - tool-call
30 - robotics
31 - ollama
32 - ternary-weights
33 - ternary-transformer
34 - quantization
35 - 1.58-bit
36 license: mit
37 ---
38
39 # JiRack Ternary Architedure
40 - Benefits high quality CPU inference TQ_2 on Llama.cpp and Ollama via QAT
41 - Robotcs, Routing, Coding, Multimedia, Advanced tool calling via JiRackDeltaNetTokenizer
42 - JiRack DeltaNet understand video and images that best for Robotics also
43
44 ## PARTNERSHIP
45 - NVIDIA
46 - FISERV
47
48 # JiRack DeltaNet 27B (CPU)
49 A fast and efficient 27B model optimized for CPU inference. Built on a Qwen3.8-style DeltaNet architecture (hybrid attention + SSM), with an updated tokenizer that includes **Routing**, **Media**, **Vision**, **Sound**, **Tool call**, and **Robotics** tags. Ready-to-run GGUF quantizations, and native Ollama support with reasoning disabled by default for fast, direct responses.
50 - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
51
52 # JiRack service options
53 - Current quantizations were done from the FP16 model.
54 - If you need custom compression or fine-tuning, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
55 - Plus double QAT via ONNX QAT.
56 - Adapt train process to avoid catastrophic forgetting with NDA
57 - Adapt train process to avoid fast plateau in training with NDA
58 - Adapts to agentic or instruct models for tool calling, using the JiRack tokenizer to enable high-quality tool calling on small models — built as a domain-specific tool expert.
59 - Deployment and scale
60
61 # Live chat on HF Space
62 - https://huggingface.co/spaces/CMSManhattan/JiRackDeltaNet_27b
63 - https://huggingface.co/spaces/CMSManhattan/JiRack-16B-Chat
64 - https://huggingface.co/spaces/CMSManhattan/JiRack-8B-Chat
65 - https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat
66 ---
67
68 ## JiRack Codding Agent IDE
69 - It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
70 - It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
71 - Web site https://www.jirack.com
72 - Eclise Plugin site https://marketplace.eclipse.org/content/jirack-coding-agent
73 - Jetbrain Plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent
74 - VS Code Plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent
75
76 ## JiRack Codding Agent IDE files
77 - Eclise Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
78 - Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix
79 - Jetbrain plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip
80 - JiRack DeskTop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip
81
82 ---
83
84
85 # Ollama production support
86 - JiRack DeltaNet 27B runs natively on Ollama with reasoning disabled by default (no forced `<think>` blocks).
87 - Available now: **`cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning`** — https://ollama.com/cmsmanhattan
88 - Runtime override also supported: `ollama run cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning --think=false`
89 - Follow fresh Ollama platform updates
90
91
92 # Spring Boot AI tool calls examples for JiRack DeltaNet series
93 - Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba
94
95 # GoEx AI tool calls examples for JiRack DeltaNet series
96 - Tool call library on python https://github.com/ShishirPatil/gorilla
97
98 # JiRack DeltaNet tool calls to boost tool call quality
99 - Use JiRack Precision tokenizer tags for tool calls with ToolBench https://github.com/OpenBMB/ToolBench
100 - https://huggingface.co/xalss/Qwen2-7B-Instruct-glaive-function-calling
101 - https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
102 - Add JiRack tool call tags in the dataset and modify tool call processor if needed
103
104
105 # JiRack RoboTech
106 - Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other
107 - [CMSManhattan/JiRackDeltaNetTokenizer](https://huggingface.co/CMSManhattan/JiRackDeltaNetTokenizer)
108
109
110 ## Available Variants on Docker with UI
111 | Tag | Quant | Size | Approx. RAM | Description |
112 |-----|-------|------|-------------|-------------|
113 | `cmsmanhattan/jirack-deltanet-27b-cpu:latest` | Full | ~55 GB | ~56–64 GB | Full precision reference |
114 | `cmsmanhattan/jirack_deltanet_27b-cpu-q4:latest` | Q4_K_M | ~16.8 GB | ~18–24 GB | Recommended balance |
115 | `cmsmanhattan/jirack-deltanet-27b-cpu-q3:latest` | Q3_K_M | ~13.9 GB | ~15–20 GB | Good quality / size trade-off |
116 | `cmsmanhattan/jirack-deltanet-27b-cpu-q2:latest` | Q2_K | ~11.2 GB | ~12–17 GB | Maximum compression |
117 | `cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest`| TQ_2 | ~7.2 GB | ~8–13 GB | Ultra compression |
118
119
120
121 ## Quick Start
122
123 ### Run with Ollama (recommended — reasoning off by default)
124 - https://ollama.com/cmsmanhattan
125 ```bash
126 ollama pull cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning
127 ollama run cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning "What is the capital of France?"
128 ```
129
130 ### Run with Docker
131 - 27B docker images can be provided by request.
132 - Build docker on local from source or request from me.
133 - Docker images use hidden reasoning logic to make chat clear
134
135
136 **Q8/int8**
137 ```bash
138 docker run -d \
139 --name jirack_deltanet_27b \
140 -p 7869:7869 \
141 --cpus=16 \
142 -e THREADS=16 \
143 -e THREADS_BATCH=16 \
144 --restart unless-stopped \
145 cmsmanhattan/jirack_deltanet_27b-cpu-q8:latest
146 ```
147
148 **Q6**
149 ```bash
150 docker run -d \
151 --name jirack_deltanet_27b \
152 -p 7869:7869 \
153 --cpus=16 \
154 -e THREADS=16 \
155 -e THREADS_BATCH=16 \
156 --restart unless-stopped \
157 cmsmanhattan/jirack_deltanet_27b-cpu-q6:latest
158 ```
159
160 **Default CPU (Q4/int4 recommended)**
161 ```bash
162 docker run -d \
163 --name jirack_deltanet_27b \
164 -p 7869:7869 \
165 --restart unless-stopped \
166 cmsmanhattan/jirack_deltanet_27b-cpu-q4:latest
167 ```
168 **Q3**
169 ```bash
170 docker run -d \
171 --name jirack_deltanet_27b \
172 -p 7869:7869 \
173 --cpus=16 \
174 -e THREADS=16 \
175 -e THREADS_BATCH=16 \
176 --restart unless-stopped \
177 cmsmanhattan/jirack_deltanet_27b-cpu-q3:latest
178 ```
179 **Q2 (lowest memory)**
180 ```bash
181 docker run -d \
182 --name jirack_deltanet_27b \
183 -p 7869:7869 \
184 --cpus=16 \
185 -e THREADS=16 \
186 -e THREADS_BATCH=16 \
187 --restart unless-stopped \
188 cmsmanhattan/jirack_deltanet_27b-cpu-q2:latest
189 ```
190
191 **TQ_2 (Ultra lowe memory)**
192 ```bash
193 docker run -d \
194 --name jirack_deltanet_27b \
195 -p 7869:7869 \
196 --cpus=16 \
197 -e THREADS=16 \
198 -e THREADS_BATCH=16 \
199 --restart unless-stopped \
200 cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest
201 ```
202
203
204
205 **Full precision**
206 ```bash
207 docker run -d \
208 --name jirack_deltanet_27b \
209 -p 7869:7869 \
210 --cpus=16 \
211 -e THREADS=16 \
212 -e THREADS_BATCH=16 \
213 --restart unless-stopped \
214 cmsmanhattan/jirack-deltanet-27b-cpu:latest
215 ```
216 **Multi CPU**
217 ```bash
218 docker run -d \
219 --name jirack_deltanet_27b \
220 -p 7869:7869 \
221 --cpus=16 \
222 -e THREADS=16 \
223 -e THREADS_BATCH=16 \
224 --restart unless-stopped \
225 --memory=32g \
226 --cpus=8 \
227 cmsmanhattan/jirack-deltanet-27b-q4:latest
228 ```
229
230 ### Docker Compose Example
231 ```yaml
232 services:
233 jirack:
234 image: cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest
235 container_name: jirack_deltanet_27b
236 ports:
237 - "7869:7869"
238 volumes:
239 - .:/app
240 - ./web:/app/web
241 environment:
242 - MAX_TOKENS=2048
243 - TEMPERATURE=0.7
244 - TOP_P=0.8
245 - TOP_K=20
246 - DEFAULT_STREAM=False
247 - INTRA_THREADS=4
248 - USE_ENV_ALLOCATOR=1
249 - THREADS=16
250 - THREADS_BATCH=16
251 deploy:
252 resources:
253 limits:
254 memory: 32g
255 ```
256
257 ## Ollama platform
258 - CPU without JiRack UI
259 - GPU without JiRack UI
260 - Ollama images displays reasoning logic by default but it can be off .
261 - Use JiRack UI from https://www.jirack.com
262 - cmsmanhattan/JiRackDeltaNet_27b-reasoning:latest 54 GB
263 - cmsmanhattan/JiRackDeltaNet_27b-q2-reasoning:latest 10 GB
264 - cmsmanhattan/JiRackDeltaNet_27b-q3-reasoning:latest 13 GB
265 - cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning:latest 16 GB
266 - cmsmanhattan/JiRackDeltaNet_27b-q6-reasoning:latest 22 GB
267 - cmsmanhattan/JiRackDeltaNet_27b-q8-reasoning:latest 29 GB
268
269
270 ## Access the UI
271 Once the container is running, open your browser and navigate to:
272 `http://localhost:7869`
273 This opens the JiRack UI — a clean web interface.
274
275 ## Changing the Port
276 The listening port can be easily modified directly from the **Settings** panel within the JiRack UI.
277
278 ## Licensing
279
280 - Model weights are released under the MIT License — free to use, modify, and distribute for any purpose, including commercial. No royalties, no per-user fees, no subscription.
281 - The Docker image with UI and the pre-built Ollama quantizations are separate paid products. If you prefer to build your own secure deployment — take the weights, assemble your own stack, and you're done.
282
283 - The JiRack DeltaNet 27B model for Docker and Ollama is provided under a commercial license ($12 per user per year).
284 - All JiRack UI clients are provided under a commercial license.
285 - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
286
287 For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.
288 - **JiRack MS Windows 11 Desktop Client (with Ollama API):**
289 https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
290 - **Live email chat with the model:** support@cmsmanhattan.com
291
292 ## Hardware Recommendations
293
294 ### Recommended Hardware for JiRack DeltaNet 27B (single Docker container)
295 | Use Case | CPU | RAM | Recommended Quant | Expected Speed | Recommendation |
296 |-------------------|----------------------------|-----------|--------------------|---------------------|----------------|
297 | Recommended | Ryzen 9 / Intel i9 / Xeon | 24–32 GB | Q4_K_M | Good interactive | Best choice |
298 | High Performance | High-core server CPU | 48 GB+ | Full / Q4 | Excellent | Excellent |
299 | Low Memory | Modern 12+ core CPU | 16–24 GB | Q3_K_M or Q2_K | Usable | Acceptable |
300 | Edge / Minimal | Strong workstation CPU | 16 GB | Q2_K | Acceptable | Budget option |
301
302 ## Important Memory Notes
303 Even though the quantized 27B models are relatively compact for their size, we recommend the following for best experience:
304 - Q4_K_M: 18–24 GB system RAM minimum
305 - Q3_K_M / Q2_K: 15–20 GB system RAM
306 - Full precision: 48 GB+ system RAM recommended
307
308 Reasons for extra headroom:
309 - KV-cache consumption during generation
310 - Runtime overhead and temporary buffers
311 - System stability and avoiding out-of-memory errors
312 - Room for larger context windows
313
314 **Minimum recommended (Q4):** 18 GB system RAM
315 **Ideal:** 24–32 GB system RAM
316
317 I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
318
319 ## Architecture Notes
320 - **Qwen3.8-style DeltaNet architecture**: hybrid attention + SSM design (`qwen35` in GGUF metadata)
321 - **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics**
322 - **No-forced-reasoning support**: chat template patched so `<think>` blocks default to closed; native support on Ollama
323 - Hidden 5120, 65 layers, attention heads 24 / KV heads 4, feed-forward 17408
324 - SSM: conv kernel 4, state size 128, group count 16, time-step rank 48, inner size 6144
325 - RoPE θ = 10,000,000, RMSNorm ε = 1e-6
326 - Context length: up to 262,144 tokens
327 - Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M)
328
329 ## Benchmarks
330 JiRack DeltaNet 27B is built on the Qwen3.5/Qwen3.8-style DeltaNet architecture. The table below reproduces the published base-model benchmark results from [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for reference — these reflect the upstream base model's capabilities, not JiRack-specific fine-tuning or quantization results.
331
332 ### Text Performance
333
334 | | Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max |
335 |---------------------------------------------------------|-------------|-------------|--------------|-------------------|-------------|
336 | **Coding** | | | | | |
337 | Agentic terminal coding — Terminal Bench 2.1 (Terminus) | 73.0 | 63.4 | 64.0 | 51.7 | **78.2** |
338 | Agentic coding — SWE-bench Pro | **61.7** | 53.5 | 57.6 | 51.2 | 53.4 |
339 | Repo-level code generation — NL2Repo-Bench | 42.3 | 36.2 | 41.1 | -- | **47.6** |
340 | Agentic coding — DeepSWE 1.1 | **42.2** | 13.3 | 14.2 | -- | -- |
341 | Software engineering — QwenSWEBench | **79.0** | 49.3 | 59.2 | -- | 63.8 |
342 | **Agent** | | | | | |
343 | Long-horizon office work — CoWorkBench | **70.7** | 61.0 | 65.1 | -- | 68.2 |
344 | Professional job tasks — JobBench | **33.4** | 21.8 | 27.6 | -- | -- |
345 | Frontier agentic tasks — Agents' Last Exam (Pass@1/Score)| **20.4 / 42.9** | 10.6 / 27.3 | 13.2 / 33.6 | -- | -- |
346 | **General** | | | | | |
347 | Instruction following — IFBench | **79.5** | 69.1 | 79.1 | 77.0 | 62.5 |
348 | Scientific reasoning — GPQA Diamond | 89.2 | 87.8 | 90.3 | 83.5 | **91.3** |
349 | Multidisciplinary reasoning — HLE | 30.8 | 24.0 | 34.7 | 22.0 | **40.0** |
350 | Competitive coding — LiveCodeBench v6 | **90.3** | 83.9 | 89.6 | -- | 88.8 |
351
352 ### VL Performance
353
354 | | Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max |
355 |------------------------------------------|-------------|-------------|--------------|-------------------|-------------|
356 | **Agentic Multimodal Intelligence** | | | | | |
357 | Computer use — OSWorld-Verified | **84.3** | 63.9 | 73.3 | 65.9 | 72.7 |
358 | Browser use — WebArena-Verified | **64.8** | 48.8 | 55.3 | -- | -- |
359 | Mobile use — AndroidWorld | **81.9** | 70.3 | 81.0 | -- | 62.0 |
360 | Application recreation — RecreationBench | **47.1** | 29.8 | 30.2 | -- | -- |
361 | Multimodal tool use — ClawEval-MM (Pass@3/Avg) | 57.4 / 56.9 | 42.6 / 50.4 | **57.4 / 60.1** | -- | 52.5 / 54.7 |
362 | Multimodal SWE — SWE-MM | **38.6** | 25.7 | 30.0 | -- | 27.1 |
363 | Visual web development — Vision2Web | **62.9** | 45.0 | 42.1 | -- | -- |
364 | **General Multimodal Intelligence** | | | | | |
365 | Visual math — MathVision (w/o CI / w/ CI)| 90.0 / **94.6** | 85.1 / -- | **90.3** / -- | -- | 65.5 / -- |
366 | General visual reasoning — BabyVision (w/o CI / w/ CI) | **65.7 / 85.6** | 28.9 / -- | 64.7 / 70.4 | -- | 12.6 / -- |
367 | Scientific chart analysis — CharXiv (RQ) (w/o CI / w/ CI) | 83.7 / **90.2** | 78.4 / -- | **85.8** / 85.9 | 78.8 | 66.0 / -- |
368 | Document intelligence — OmniDocBench 1.5 | 91.1 | 89.4 | **91.4** | 75.8 | 86.6 |
369 | Real-world perception — RealWorldQA | 85.9 | 84.1 | **86.9** | -- | 73.9 |
370 | Embodied intelligence — ERQA | 65.5 | 62.5 | **69.8** | -- | 40.8 |
371
372 *Source: [Qwen/Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B). Best result in each row is bolded. Empty cells (--) indicate results not yet available. See the source card for full evaluation methodology and footnotes.*
373
374 ## 📧 Contact & Licensing
375 For joint venture opportunities, hardware integration, or licensing inquiries:
376 - **Email:** grabko@cmsmanhattan.com
377 - **Phone:** +1 (516) 777-0945
378 - **Location:** New York, USA
379
380
381 ## License
382 MIT License
383
384 ![NVIDIA Inception Program Member](https://huggingface.co/spaces/CMSManhattan/README/resolve/main/badge.jpg)
385