README.md
10.4 KB · 275 lines · markdown Raw
1 ---
2 language:
3 - en
4 - zh
5 - ja
6 - ko
7 - fr
8 - es
9 - pt
10 - de
11 - it
12 - ru
13 - ar
14 - vi
15 - th
16 tags:
17 - text-generation
18 - ternary
19 - bitnet
20 - 1.58bit
21 - cpu
22 - gguf
23 - deepseek
24 - efficient
25 - low-memory
26 - jirack
27 - web-ui
28 - routing
29 - tool-call
30 - robotics
31 license: mit
32 ---
33 # JiRack Ultra 1B (CPU)
34 A fast and efficient ~1.5B model optimized for CPU inference. The model was refactored with BitNet features and an updated tokenizer that includes new **Routing**, **Tool call**, and **Robotics** tags. Built on a DeepSeek model redesigned by CMSManhattan with native ternary (BitNet-style) support and ready-to-run GGUF quantizations.
35 - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
36
37
38 # JiRack Ternary Architedure & JiRack Tokenizer
39 - Benefits high quality CPU inference TQ_2 on Llama.cpp and Ollama via QAT
40 - Robotcs, Routing, Coding, Multimedia, Advanced tool calling via CMSManhattan/JiRackPrecisionTokenizer
41
42 ## PARTNERSHIP
43
44 - NVIDIA
45 - FISERV
46
47 # Ollama production support
48 - We are working to support JiRack on Ollama for production systems also
49 - added Jirack chat without reasoning feature https://ollama.com/cmsmanhattan
50 - Follow fresh Ollama platform updates
51
52 # JiRack sevice options
53 - Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture.
54 - If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
55 - Plus double QAT via ONNX QAT.
56 - Adapt train process to avoid catastrophic forgetting with NDA
57 - Adapt train process to avoid fast plato in training with NDA
58 - Convert model to TQ2_0 with support AVX2 and AVX-512 CPU instructions for high performance on CPU
59 - QAT for TQ_2 Llama.cpp Ternarization docs https://huggingface.co/CMSManhattan/JiRackUltra_1b/blob/main/QAT_to_Llama.cpp_GGUF_TQ2_0_JirackUltra_1b.md
60 - Adapts to agentic or instruct models for tool calling, using the JiRak tokenizer to enable high-quality tool calling on small models — built as a domain-specific tool expert.
61 - Deployment and scale
62 - JiRack Ternary production demo: https://huggingface.co/CMSManhattan/JiRackTernaryPrecision-2B-chat
63
64
65 ## Live chat
66
67 - https://huggingface.co/spaces/CMSManhattan/JiRackUltra_1b
68
69 ## Live chat TQ2_O quants
70 - https://huggingface.co/spaces/CMSManhattan/JiRack-36B-Chat
71 - https://huggingface.co/spaces/CMSManhattan/JiRack-16B-Chat
72 - https://huggingface.co/spaces/CMSManhattan/JiRack-8B-Chat
73 - https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat
74
75 ---
76
77 ## JiRack Codding Agent IDE
78 - It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
79 - It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
80 - Web site https://www.jirack.com
81 - Eclise Plugin site https://marketplace.eclipse.org/content/jirack-coding-agent
82 - Jetbrain Plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent
83 - VS Code Plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent
84
85 ## JiRack Codding Agent IDE files
86 - Eclise Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
87 - Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix
88 - Jetbrain plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip
89 - JiRack DeskTop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip
90
91 ---
92
93
94
95
96
97 # Spring Boot AI tool calls examples for JiRack Ultra series
98 - Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba
99
100 # GoEx AI tool calls examples for JiRack Ultra series
101 - Tool call library on python https://github.com/ShishirPatil/gorilla
102
103 # JiRack Ultra 1 tool calls to boost tool call quality
104 - Use JiRack Precision tokenzer tags for tool calls with ToolBench https://github.com/OpenBMB/ToolBench
105 - https://huggingface.co/xalss/Qwen2-7B-Instruct-glaive-function-calling
106 - https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
107 - Add JiRack tool call tags in the dataset and modify tool call processor if needed
108
109
110
111
112 # JiRack RoboTech
113 - Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other
114 - [CMSManhattan/JiRackPrecisionTokenizer](https://huggingface.co/CMSManhattan/JiRackPrecisionTokenizer)
115
116 ## Export to ONNX
117
118 - ONNX guide https://huggingface.co/CMSManhattan/JiRackUltra_1b/blob/main/jirackultra_1b_onnx_export_guide.md
119
120
121
122 ## Available Variants
123 | Tag | Quant | Size | Approx. RAM | Description |
124 |-----|-------|------|-------------|-------------|
125 | `cmsmanhattan/jirack-ultra-1b-cpu:latest` | Full | 3.56 GB | ~4.1 GB | Full ternary reference |
126 | `cmsmanhattan/jirack-ultra-1b-cpu-q4:latest` | Q4_K_M | 1.12 GB | ~1.6 GB | Recommended balance |
127 | `cmsmanhattan/jirack-ultra-1b-cpu-q3:latest` | Q3_K_M | 0.92 GB | ~1.4 GB | Good quality / size trade-off |
128 | `cmsmanhattan/jirack-ultra-1b-cpu-q2:latest` | Q2_K | 0.24 GB | ~1.0 GB | Maximum compression |
129 ## Quick Start
130 ### Run with Docker
131 **Default CPU (Q4 recommended)**
132 ```bash
133 docker run -d \
134 --name jirack_ultra_1b \
135 -p 7869:7869 \
136 --cpus=16 \
137 -e THREADS=16 \
138 -e THREADS_BATCH=16 \
139 --restart unless-stopped \
140 cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
141 ```
142 **Q3**
143 ```bash
144 docker run -d \
145 --name jirack_ultra_1b \
146 -p 7869:7869 \
147 --cpus=16 \
148 -e THREADS=16 \
149 -e THREADS_BATCH=16 \
150 --restart unless-stopped \
151 cmsmanhattan/jirack-ultra-1b-cpu-q3:latest
152 ```
153 **Q2 (lowest memory)**
154 ```bash
155 docker run -d \
156 --name jirack_ultra_1b \
157 -p 7869:7869 \
158 --cpus=16 \
159 -e THREADS=16 \
160 -e THREADS_BATCH=16 \
161 --restart unless-stopped \
162 cmsmanhattan/jirack-ultra-1b-cpu-q2:latest
163 ```
164 **Full precision**
165 ```bash
166 docker run -d \
167 --name jirack_ultra_1b \
168 -p 7869:7869 \
169 --cpus=16 \
170 -e THREADS=16 \
171 -e THREADS_BATCH=16 \
172 --restart unless-stopped \
173 cmsmanhattan/jirack-ultra-1b-cpu:latest
174 ```
175 **Multi CPU**
176 ```bash
177 docker run -d \
178 --name jirack_ultra_1b \
179 -p 7869:7869 \
180 --cpus=16 \
181 -e THREADS=16 \
182 -e THREADS_BATCH=16 \
183 --restart unless-stopped \
184 --memory=4g \
185 --cpus=4 \
186 cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
187 ```
188 ### Docker Compose Example
189 ```yaml
190 services:
191 jirack:
192 image: cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
193 container_name: jirack_ultra_1b
194 ports:
195 - "7869:7869"
196 volumes:
197 - .:/app
198 - ./web:/app/web
199 environment:
200 - MAX_TOKENS=2048
201 - TEMPERATURE=0.7
202 - TOP_P=0.9
203 - DEFAULT_STREAM=False
204 - INTRA_THREADS=4
205 - USE_ENV_ALLOCATOR=1
206 - THREADS=16
207 - THREADS_BATCH=16
208 deploy:
209 resources:
210 limits:
211 memory: 4g
212 ```
213 ## Access the UI
214 Once the container is running, open your browser and navigate to:
215 `http://localhost:7869`
216 This opens the JiRack UI — a clean web interface.
217 ## Changing the Port
218 The listening port can be easily modified directly from the **Settings** panel within the JiRack UI.
219
220 ## Licensing
221
222 - Model weights are released under the MIT License — free to use, modify, and distribute for any purpose, including commercial. No royalties, no per-user fees, no subscription.
223 - The Docker image with UI and the pre-built Ollama quantizations are separate paid products. If you prefer to build your own secure deployment — take the weights, assemble your own stack, and you're done.
224 - The JiRack Ultra 1B model for Docker and Ollama is provided under a commercial license ($12 per user per year).
225 - All JiRack UI clients are provided under a commercial license.
226 - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
227 For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.
228 - **JiRack MS Windows 11 Desktop Client (with Ollama API):**
229 https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
230 - **Live email chat with the model:** support@cmsmanhattan.com
231 ## Hardware Recommendations
232 ### Recommended Hardware for JiRack Ultra 1B (single Docker container)
233 | Use Case | CPU | RAM | Recommended Quant | Expected Speed | Recommendation |
234 |-------------------|------------------------------|----------|-------------------|---------------------|----------------|
235 | Recommended | Ryzen 5 / Intel i5 | 4–8 GB | Q4_K_M | Excellent interactive | Best choice |
236 | High Performance | Ryzen 7 / Intel i7 | 8–16 GB | Full / Q4 | Excellent | Excellent |
237 | Low Memory | Modern 4+ core CPU | 2–4 GB | Q3_K_M or Q2_K | Usable | Acceptable |
238 | Edge / Minimal | Laptop / SBC CPU | 2 GB | Q2_K | Acceptable | Budget option |
239 ## Important Memory Notes
240 Even though the quantized 1B models are very small, we recommend the following for best experience:
241 - Q4_K_M: 2–4 GB system RAM minimum
242 - Q3_K_M / Q2_K: 1.5–3 GB system RAM
243 - Full precision: 3–4 GB+ system RAM recommended
244 Reasons for extra headroom:
245 - KV-cache consumption during generation
246 - Runtime overhead and temporary buffers
247 - System stability and avoiding out-of-memory errors
248 - Room for larger context windows
249 **Minimum recommended (Q4):** 2–3 GB system RAM
250 **Ideal:** 4–8 GB system RAM
251 I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
252 ## Architecture Notes
253 - **Refactored with BitNet features**: Native BitLinear ternary path (b1.58-style) with λ-warmup STE
254 - **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics**
255 - Architecture: Qwen2 layout (hidden 1,536, intermediate 8,960, 28 layers, GQA 12/2, vocabulary 151,936, untied embeddings, RoPE theta 10,000)
256 - RoPE θ = 10000, RMSNorm ε = 1e-6
257 - Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M)
258
259 ## ROBOS HUMANOID Project
260
261 - https://github.com/cmsmanhattan/ROBOS
262
263 ## 📧 Contact & Licensing
264 For joint venture opportunities, hardware integration, or licensing inquiries:
265 - **Email:** grabko@cmsmanhattan.com
266 - **Phone:** +1 (516) 777-0945
267 - **Location:** New York, USA
268
269 ## License
270 MIT License
271
272 Base models: a DeepSeek model (MIT License, `LICENSE-DEEPSEEK`), itself derived from a Qwen model (Apache License 2.0, `LICENSE-APACHE`); see `NOTICE.md`.
273
274 ![NVIDIA Inception Program Member](https://huggingface.co/spaces/CMSManhattan/README/resolve/main/badge.jpg)
275