README.md
| 1 | --- |
| 2 | language: |
| 3 | - en |
| 4 | - zh |
| 5 | - ja |
| 6 | - ko |
| 7 | - fr |
| 8 | - es |
| 9 | - pt |
| 10 | - de |
| 11 | - it |
| 12 | - ru |
| 13 | - ar |
| 14 | - vi |
| 15 | - th |
| 16 | tags: |
| 17 | - text-generation |
| 18 | - deltanet |
| 19 | - qwen3.8 |
| 20 | - gguf |
| 21 | - cpu |
| 22 | - efficient |
| 23 | - low-memory |
| 24 | - ternary |
| 25 | - bitnet |
| 26 | - jirack |
| 27 | - web-ui |
| 28 | - routing |
| 29 | - tool-call |
| 30 | - robotics |
| 31 | - ollama |
| 32 | - ternary-weights |
| 33 | - ternary-transformer |
| 34 | - quantization |
| 35 | - 1.58-bit |
| 36 | license: mit |
| 37 | --- |
| 38 | |
| 39 | # JiRack Ternary Architedure |
| 40 | - Benefits high quality CPU inference TQ_2 on Llama.cpp and Ollama via QAT |
| 41 | - Robotcs, Routing, Coding, Multimedia, Advanced tool calling via JiRackDeltaNetTokenizer |
| 42 | - JiRack DeltaNet understand video and images that best for Robotics also |
| 43 | |
| 44 | ## PARTNERSHIP |
| 45 | - NVIDIA |
| 46 | - FISERV |
| 47 | |
| 48 | # JiRack DeltaNet 27B (CPU) |
| 49 | A fast and efficient 27B model optimized for CPU inference. Built on a Qwen3.8-style DeltaNet architecture (hybrid attention + SSM), with an updated tokenizer that includes **Routing**, **Media**, **Vision**, **Sound**, **Tool call**, and **Robotics** tags. Ready-to-run GGUF quantizations, and native Ollama support with reasoning disabled by default for fast, direct responses. |
| 50 | - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative. |
| 51 | |
| 52 | # JiRack service options |
| 53 | - Current quantizations were done from the FP16 model. |
| 54 | - If you need custom compression or fine-tuning, please write to me and I'll perform QAT from your dataset, tailored specifically to your task. |
| 55 | - Plus double QAT via ONNX QAT. |
| 56 | - Adapt train process to avoid catastrophic forgetting with NDA |
| 57 | - Adapt train process to avoid fast plateau in training with NDA |
| 58 | - Adapts to agentic or instruct models for tool calling, using the JiRack tokenizer to enable high-quality tool calling on small models — built as a domain-specific tool expert. |
| 59 | - Deployment and scale |
| 60 | |
| 61 | # Live chat on HF Space |
| 62 | - https://huggingface.co/spaces/CMSManhattan/JiRackDeltaNet_27b |
| 63 | - https://huggingface.co/spaces/CMSManhattan/JiRack-16B-Chat |
| 64 | - https://huggingface.co/spaces/CMSManhattan/JiRack-8B-Chat |
| 65 | - https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat |
| 66 | --- |
| 67 | |
| 68 | ## JiRack Codding Agent IDE |
| 69 | - It is Agent Coding IDE for JiRack Models to run via Ollama on home PC |
| 70 | - It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review. |
| 71 | - Web site https://www.jirack.com |
| 72 | - Eclise Plugin site https://marketplace.eclipse.org/content/jirack-coding-agent |
| 73 | - Jetbrain Plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent |
| 74 | - VS Code Plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent |
| 75 | |
| 76 | ## JiRack Codding Agent IDE files |
| 77 | - Eclise Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip |
| 78 | - Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix |
| 79 | - Jetbrain plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip |
| 80 | - JiRack DeskTop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip |
| 81 | |
| 82 | --- |
| 83 | |
| 84 | |
| 85 | # Ollama production support |
| 86 | - JiRack DeltaNet 27B runs natively on Ollama with reasoning disabled by default (no forced `<think>` blocks). |
| 87 | - Available now: **`cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning`** — https://ollama.com/cmsmanhattan |
| 88 | - Runtime override also supported: `ollama run cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning --think=false` |
| 89 | - Follow fresh Ollama platform updates |
| 90 | |
| 91 | |
| 92 | # Spring Boot AI tool calls examples for JiRack DeltaNet series |
| 93 | - Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba |
| 94 | |
| 95 | # GoEx AI tool calls examples for JiRack DeltaNet series |
| 96 | - Tool call library on python https://github.com/ShishirPatil/gorilla |
| 97 | |
| 98 | # JiRack DeltaNet tool calls to boost tool call quality |
| 99 | - Use JiRack Precision tokenizer tags for tool calls with ToolBench https://github.com/OpenBMB/ToolBench |
| 100 | - https://huggingface.co/xalss/Qwen2-7B-Instruct-glaive-function-calling |
| 101 | - https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1 |
| 102 | - Add JiRack tool call tags in the dataset and modify tool call processor if needed |
| 103 | |
| 104 | |
| 105 | # JiRack RoboTech |
| 106 | - Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other |
| 107 | - [CMSManhattan/JiRackDeltaNetTokenizer](https://huggingface.co/CMSManhattan/JiRackDeltaNetTokenizer) |
| 108 | |
| 109 | |
| 110 | ## Available Variants on Docker with UI |
| 111 | | Tag | Quant | Size | Approx. RAM | Description | |
| 112 | |-----|-------|------|-------------|-------------| |
| 113 | | `cmsmanhattan/jirack-deltanet-27b-cpu:latest` | Full | ~55 GB | ~56–64 GB | Full precision reference | |
| 114 | | `cmsmanhattan/jirack_deltanet_27b-cpu-q4:latest` | Q4_K_M | ~16.8 GB | ~18–24 GB | Recommended balance | |
| 115 | | `cmsmanhattan/jirack-deltanet-27b-cpu-q3:latest` | Q3_K_M | ~13.9 GB | ~15–20 GB | Good quality / size trade-off | |
| 116 | | `cmsmanhattan/jirack-deltanet-27b-cpu-q2:latest` | Q2_K | ~11.2 GB | ~12–17 GB | Maximum compression | |
| 117 | | `cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest`| TQ_2 | ~7.2 GB | ~8–13 GB | Ultra compression | |
| 118 | |
| 119 | |
| 120 | |
| 121 | ## Quick Start |
| 122 | |
| 123 | ### Run with Ollama (recommended — reasoning off by default) |
| 124 | - https://ollama.com/cmsmanhattan |
| 125 | ```bash |
| 126 | ollama pull cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning |
| 127 | ollama run cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning "What is the capital of France?" |
| 128 | ``` |
| 129 | |
| 130 | ### Run with Docker |
| 131 | - 27B docker images can be provided by request. |
| 132 | - Build docker on local from source or request from me. |
| 133 | - Docker images use hidden reasoning logic to make chat clear |
| 134 | |
| 135 | |
| 136 | **Q8/int8** |
| 137 | ```bash |
| 138 | docker run -d \ |
| 139 | --name jirack_deltanet_27b \ |
| 140 | -p 7869:7869 \ |
| 141 | --cpus=16 \ |
| 142 | -e THREADS=16 \ |
| 143 | -e THREADS_BATCH=16 \ |
| 144 | --restart unless-stopped \ |
| 145 | cmsmanhattan/jirack_deltanet_27b-cpu-q8:latest |
| 146 | ``` |
| 147 | |
| 148 | **Q6** |
| 149 | ```bash |
| 150 | docker run -d \ |
| 151 | --name jirack_deltanet_27b \ |
| 152 | -p 7869:7869 \ |
| 153 | --cpus=16 \ |
| 154 | -e THREADS=16 \ |
| 155 | -e THREADS_BATCH=16 \ |
| 156 | --restart unless-stopped \ |
| 157 | cmsmanhattan/jirack_deltanet_27b-cpu-q6:latest |
| 158 | ``` |
| 159 | |
| 160 | **Default CPU (Q4/int4 recommended)** |
| 161 | ```bash |
| 162 | docker run -d \ |
| 163 | --name jirack_deltanet_27b \ |
| 164 | -p 7869:7869 \ |
| 165 | --restart unless-stopped \ |
| 166 | cmsmanhattan/jirack_deltanet_27b-cpu-q4:latest |
| 167 | ``` |
| 168 | **Q3** |
| 169 | ```bash |
| 170 | docker run -d \ |
| 171 | --name jirack_deltanet_27b \ |
| 172 | -p 7869:7869 \ |
| 173 | --cpus=16 \ |
| 174 | -e THREADS=16 \ |
| 175 | -e THREADS_BATCH=16 \ |
| 176 | --restart unless-stopped \ |
| 177 | cmsmanhattan/jirack_deltanet_27b-cpu-q3:latest |
| 178 | ``` |
| 179 | **Q2 (lowest memory)** |
| 180 | ```bash |
| 181 | docker run -d \ |
| 182 | --name jirack_deltanet_27b \ |
| 183 | -p 7869:7869 \ |
| 184 | --cpus=16 \ |
| 185 | -e THREADS=16 \ |
| 186 | -e THREADS_BATCH=16 \ |
| 187 | --restart unless-stopped \ |
| 188 | cmsmanhattan/jirack_deltanet_27b-cpu-q2:latest |
| 189 | ``` |
| 190 | |
| 191 | **TQ_2 (Ultra lowe memory)** |
| 192 | ```bash |
| 193 | docker run -d \ |
| 194 | --name jirack_deltanet_27b \ |
| 195 | -p 7869:7869 \ |
| 196 | --cpus=16 \ |
| 197 | -e THREADS=16 \ |
| 198 | -e THREADS_BATCH=16 \ |
| 199 | --restart unless-stopped \ |
| 200 | cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest |
| 201 | ``` |
| 202 | |
| 203 | |
| 204 | |
| 205 | **Full precision** |
| 206 | ```bash |
| 207 | docker run -d \ |
| 208 | --name jirack_deltanet_27b \ |
| 209 | -p 7869:7869 \ |
| 210 | --cpus=16 \ |
| 211 | -e THREADS=16 \ |
| 212 | -e THREADS_BATCH=16 \ |
| 213 | --restart unless-stopped \ |
| 214 | cmsmanhattan/jirack-deltanet-27b-cpu:latest |
| 215 | ``` |
| 216 | **Multi CPU** |
| 217 | ```bash |
| 218 | docker run -d \ |
| 219 | --name jirack_deltanet_27b \ |
| 220 | -p 7869:7869 \ |
| 221 | --cpus=16 \ |
| 222 | -e THREADS=16 \ |
| 223 | -e THREADS_BATCH=16 \ |
| 224 | --restart unless-stopped \ |
| 225 | --memory=32g \ |
| 226 | --cpus=8 \ |
| 227 | cmsmanhattan/jirack-deltanet-27b-q4:latest |
| 228 | ``` |
| 229 | |
| 230 | ### Docker Compose Example |
| 231 | ```yaml |
| 232 | services: |
| 233 | jirack: |
| 234 | image: cmsmanhattan/jirack-deltanet-27b-cpu-tq2:latest |
| 235 | container_name: jirack_deltanet_27b |
| 236 | ports: |
| 237 | - "7869:7869" |
| 238 | volumes: |
| 239 | - .:/app |
| 240 | - ./web:/app/web |
| 241 | environment: |
| 242 | - MAX_TOKENS=2048 |
| 243 | - TEMPERATURE=0.7 |
| 244 | - TOP_P=0.8 |
| 245 | - TOP_K=20 |
| 246 | - DEFAULT_STREAM=False |
| 247 | - INTRA_THREADS=4 |
| 248 | - USE_ENV_ALLOCATOR=1 |
| 249 | - THREADS=16 |
| 250 | - THREADS_BATCH=16 |
| 251 | deploy: |
| 252 | resources: |
| 253 | limits: |
| 254 | memory: 32g |
| 255 | ``` |
| 256 | |
| 257 | ## Ollama platform |
| 258 | - CPU without JiRack UI |
| 259 | - GPU without JiRack UI |
| 260 | - Ollama images displays reasoning logic by default but it can be off . |
| 261 | - Use JiRack UI from https://www.jirack.com |
| 262 | - cmsmanhattan/JiRackDeltaNet_27b-reasoning:latest 54 GB |
| 263 | - cmsmanhattan/JiRackDeltaNet_27b-q2-reasoning:latest 10 GB |
| 264 | - cmsmanhattan/JiRackDeltaNet_27b-q3-reasoning:latest 13 GB |
| 265 | - cmsmanhattan/JiRackDeltaNet_27b-q4-reasoning:latest 16 GB |
| 266 | - cmsmanhattan/JiRackDeltaNet_27b-q6-reasoning:latest 22 GB |
| 267 | - cmsmanhattan/JiRackDeltaNet_27b-q8-reasoning:latest 29 GB |
| 268 | |
| 269 | |
| 270 | ## Access the UI |
| 271 | Once the container is running, open your browser and navigate to: |
| 272 | `http://localhost:7869` |
| 273 | This opens the JiRack UI — a clean web interface. |
| 274 | |
| 275 | ## Changing the Port |
| 276 | The listening port can be easily modified directly from the **Settings** panel within the JiRack UI. |
| 277 | |
| 278 | ## Licensing |
| 279 | |
| 280 | - Model weights are released under the MIT License — free to use, modify, and distribute for any purpose, including commercial. No royalties, no per-user fees, no subscription. |
| 281 | - The Docker image with UI and the pre-built Ollama quantizations are separate paid products. If you prefer to build your own secure deployment — take the weights, assemble your own stack, and you're done. |
| 282 | |
| 283 | - The JiRack DeltaNet 27B model for Docker and Ollama is provided under a commercial license ($12 per user per year). |
| 284 | - All JiRack UI clients are provided under a commercial license. |
| 285 | - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately. |
| 286 | |
| 287 | For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us. |
| 288 | - **JiRack MS Windows 11 Desktop Client (with Ollama API):** |
| 289 | https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip |
| 290 | - **Live email chat with the model:** support@cmsmanhattan.com |
| 291 | |
| 292 | ## Hardware Recommendations |
| 293 | |
| 294 | ### Recommended Hardware for JiRack DeltaNet 27B (single Docker container) |
| 295 | | Use Case | CPU | RAM | Recommended Quant | Expected Speed | Recommendation | |
| 296 | |-------------------|----------------------------|-----------|--------------------|---------------------|----------------| |
| 297 | | Recommended | Ryzen 9 / Intel i9 / Xeon | 24–32 GB | Q4_K_M | Good interactive | Best choice | |
| 298 | | High Performance | High-core server CPU | 48 GB+ | Full / Q4 | Excellent | Excellent | |
| 299 | | Low Memory | Modern 12+ core CPU | 16–24 GB | Q3_K_M or Q2_K | Usable | Acceptable | |
| 300 | | Edge / Minimal | Strong workstation CPU | 16 GB | Q2_K | Acceptable | Budget option | |
| 301 | |
| 302 | ## Important Memory Notes |
| 303 | Even though the quantized 27B models are relatively compact for their size, we recommend the following for best experience: |
| 304 | - Q4_K_M: 18–24 GB system RAM minimum |
| 305 | - Q3_K_M / Q2_K: 15–20 GB system RAM |
| 306 | - Full precision: 48 GB+ system RAM recommended |
| 307 | |
| 308 | Reasons for extra headroom: |
| 309 | - KV-cache consumption during generation |
| 310 | - Runtime overhead and temporary buffers |
| 311 | - System stability and avoiding out-of-memory errors |
| 312 | - Room for larger context windows |
| 313 | |
| 314 | **Minimum recommended (Q4):** 18 GB system RAM |
| 315 | **Ideal:** 24–32 GB system RAM |
| 316 | |
| 317 | I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance. |
| 318 | |
| 319 | ## Architecture Notes |
| 320 | - **Qwen3.8-style DeltaNet architecture**: hybrid attention + SSM design (`qwen35` in GGUF metadata) |
| 321 | - **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics** |
| 322 | - **No-forced-reasoning support**: chat template patched so `<think>` blocks default to closed; native support on Ollama |
| 323 | - Hidden 5120, 65 layers, attention heads 24 / KV heads 4, feed-forward 17408 |
| 324 | - SSM: conv kernel 4, state size 128, group count 16, time-step rank 48, inner size 6144 |
| 325 | - RoPE θ = 10,000,000, RMSNorm ε = 1e-6 |
| 326 | - Context length: up to 262,144 tokens |
| 327 | - Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M) |
| 328 | |
| 329 | ## Benchmarks |
| 330 | JiRack DeltaNet 27B is built on the Qwen3.5/Qwen3.8-style DeltaNet architecture. The table below reproduces the published base-model benchmark results from [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for reference — these reflect the upstream base model's capabilities, not JiRack-specific fine-tuning or quantization results. |
| 331 | |
| 332 | ### Text Performance |
| 333 | |
| 334 | | | Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max | |
| 335 | |---------------------------------------------------------|-------------|-------------|--------------|-------------------|-------------| |
| 336 | | **Coding** | | | | | | |
| 337 | | Agentic terminal coding — Terminal Bench 2.1 (Terminus) | 73.0 | 63.4 | 64.0 | 51.7 | **78.2** | |
| 338 | | Agentic coding — SWE-bench Pro | **61.7** | 53.5 | 57.6 | 51.2 | 53.4 | |
| 339 | | Repo-level code generation — NL2Repo-Bench | 42.3 | 36.2 | 41.1 | -- | **47.6** | |
| 340 | | Agentic coding — DeepSWE 1.1 | **42.2** | 13.3 | 14.2 | -- | -- | |
| 341 | | Software engineering — QwenSWEBench | **79.0** | 49.3 | 59.2 | -- | 63.8 | |
| 342 | | **Agent** | | | | | | |
| 343 | | Long-horizon office work — CoWorkBench | **70.7** | 61.0 | 65.1 | -- | 68.2 | |
| 344 | | Professional job tasks — JobBench | **33.4** | 21.8 | 27.6 | -- | -- | |
| 345 | | Frontier agentic tasks — Agents' Last Exam (Pass@1/Score)| **20.4 / 42.9** | 10.6 / 27.3 | 13.2 / 33.6 | -- | -- | |
| 346 | | **General** | | | | | | |
| 347 | | Instruction following — IFBench | **79.5** | 69.1 | 79.1 | 77.0 | 62.5 | |
| 348 | | Scientific reasoning — GPQA Diamond | 89.2 | 87.8 | 90.3 | 83.5 | **91.3** | |
| 349 | | Multidisciplinary reasoning — HLE | 30.8 | 24.0 | 34.7 | 22.0 | **40.0** | |
| 350 | | Competitive coding — LiveCodeBench v6 | **90.3** | 83.9 | 89.6 | -- | 88.8 | |
| 351 | |
| 352 | ### VL Performance |
| 353 | |
| 354 | | | Qwen3.8-27B | Qwen3.6-27B | Qwen3.7-Plus | Muse Glimmer-30B | Opus4.6 Max | |
| 355 | |------------------------------------------|-------------|-------------|--------------|-------------------|-------------| |
| 356 | | **Agentic Multimodal Intelligence** | | | | | | |
| 357 | | Computer use — OSWorld-Verified | **84.3** | 63.9 | 73.3 | 65.9 | 72.7 | |
| 358 | | Browser use — WebArena-Verified | **64.8** | 48.8 | 55.3 | -- | -- | |
| 359 | | Mobile use — AndroidWorld | **81.9** | 70.3 | 81.0 | -- | 62.0 | |
| 360 | | Application recreation — RecreationBench | **47.1** | 29.8 | 30.2 | -- | -- | |
| 361 | | Multimodal tool use — ClawEval-MM (Pass@3/Avg) | 57.4 / 56.9 | 42.6 / 50.4 | **57.4 / 60.1** | -- | 52.5 / 54.7 | |
| 362 | | Multimodal SWE — SWE-MM | **38.6** | 25.7 | 30.0 | -- | 27.1 | |
| 363 | | Visual web development — Vision2Web | **62.9** | 45.0 | 42.1 | -- | -- | |
| 364 | | **General Multimodal Intelligence** | | | | | | |
| 365 | | Visual math — MathVision (w/o CI / w/ CI)| 90.0 / **94.6** | 85.1 / -- | **90.3** / -- | -- | 65.5 / -- | |
| 366 | | General visual reasoning — BabyVision (w/o CI / w/ CI) | **65.7 / 85.6** | 28.9 / -- | 64.7 / 70.4 | -- | 12.6 / -- | |
| 367 | | Scientific chart analysis — CharXiv (RQ) (w/o CI / w/ CI) | 83.7 / **90.2** | 78.4 / -- | **85.8** / 85.9 | 78.8 | 66.0 / -- | |
| 368 | | Document intelligence — OmniDocBench 1.5 | 91.1 | 89.4 | **91.4** | 75.8 | 86.6 | |
| 369 | | Real-world perception — RealWorldQA | 85.9 | 84.1 | **86.9** | -- | 73.9 | |
| 370 | | Embodied intelligence — ERQA | 65.5 | 62.5 | **69.8** | -- | 40.8 | |
| 371 | |
| 372 | *Source: [Qwen/Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B). Best result in each row is bolded. Empty cells (--) indicate results not yet available. See the source card for full evaluation methodology and footnotes.* |
| 373 | |
| 374 | ## 📧 Contact & Licensing |
| 375 | For joint venture opportunities, hardware integration, or licensing inquiries: |
| 376 | - **Email:** grabko@cmsmanhattan.com |
| 377 | - **Phone:** +1 (516) 777-0945 |
| 378 | - **Location:** New York, USA |
| 379 | |
| 380 | |
| 381 | ## License |
| 382 | MIT License |
| 383 | |
| 384 |  |
| 385 | |