README.md · OTel-2.0-LLM-31B-IT

1

---

2

license: apache-2.0

3

license_link: https://ai.google.dev/gemma/docs/gemma_4_license

4

language:

5

- en

6

base_model:

7

- google/gemma-4-31b-it

8

base_model_relation: finetune

9

tags:

10

- telecom

11

- telecommunications

12

- gsma

13

- at-t

14

- microsoft

15

- dell

16

- amd

17

- red-hat

18

- open-telco-ai

19

- rag

20

- instruction-following

21

- tool-calling

22

- osft

23

- orthogonal-subspace-fine-tuning

24

- continual-learning

25

- domain-adaptation

26

- post-trained

27

- transformers

28

- vllm

29

- multimodal

30

pipeline_tag: text-generation

31

---

32

33

# OTel-2.0-LLM-31B-IT

34

35

> **Versioning notice:** This card describes the current OTel 2.0 OSFT checkpoint. The model weights may be updated over time. For reproducible evaluation or production deployment, pin a specific model revision, checkpoint hash, or release tag rather than relying on the floating latest version.

36

37

**OTel-2.0-LLM-31B-IT** is a telecom-specialized instruction model post-trained from **Gemma 4 31B-IT** on approximately **440 billion telecom training tokens**. It is the first release in the OTel 2.0 family and is designed to support telco-grade AI workflows across network operations, standards interpretation, product development, network configuration assistance, RAG, and telecom-specific question answering.

38

39

OTel 2.0 extends the original OTel effort from a RAG-oriented telecom fine-tuning release into a larger domain-adapted training program. The model was trained from a much larger standards and telecom corpus, with new data preparation coverage for direct telecom QnA, abstention, RAG, base-model-style telecom data, and general-purpose instruction-following and tool-calling examples. The current training mixture does not include telecommunications-specific MCP, tool-calling, or instruction-following examples.

40

41

## Release Status

42

43

This repository contains the OTel 2.0 OSFT checkpoint. Its weights may be updated as model development continues. Use a pinned model revision, checkpoint hash, or release tag for reproducible evaluation and production deployment.

44

45

## Model Details

46

47

| Attribute | Value |

48

|---|---|

49

| Base model | Gemma 4 31B-IT |

50

| Parameters | 31B |

51

| Model family | OTel 2.0 |

52

| Training method | Telecom domain post-training via **OSFT** (Orthogonal Subspace Fine-Tuning, Red Hat AI Innovation Team) |

53

| OSFT configuration | `unfreeze_rank_ratio=0.35`, 2 epochs, effective batch size 512, lr 2e-5, cosine schedule, max sequence length 4096 |

54

| Released format | Merged bf16 HuggingFace checkpoint (`Gemma4ForConditionalGeneration`), ~62.6 GB |

55

| Modality | Architecturally multimodal (text + image); **trained modality is text only** — see [Vision and Modality](#vision-and-modality) |

56

| Raw telecom corpus | ~15B tokens from GSMA through Open Telco AI |

57

| Processed data volume | >1T tokens processed using Red Hat's open-source Synthetic Data Generation Hub (SDG Hub) |

58

| Training tokens | ~440B |

59

| Compute for data processing | ~530 GPUs through Microsoft Azure Managed Compute, primarily AMD MI300X |

60

| Model training infrastructure | On-premises AMD MI355X GPUs with Dell Technologies infrastructure and servers |

61

| Day 0 inference availability | Microsoft Foundry, Featherless AI, and Red Hat |

62

| Supported interaction | English-language text; image input structurally supported but not domain-tuned |

63

64

**Repository format:** `safetensors`, `bfloat16`, 15 shards, architecture

65

`Gemma4ForConditionalGeneration` — 832 language tensors, 355 `vision_tower` tensors, and the

66

`embed_vision` projector (1,188 total). See [Vision and Modality](#vision-and-modality) for

67

what the vision path is and is not.

68

69

## Model Lineage

70

71

`Gemma 4 31B-IT` -> OTel 2.0 telecom data processing -> OSFT post-training (orthogonal subspace, `unfreeze_rank_ratio=0.35`) -> `OTel-2.0-LLM-31B-IT`

72

73

## OSFT Training Details

74

75

This checkpoint was produced with **OSFT (Orthogonal Subspace Fine-Tuning)**, the continual-learning

76

algorithm from the Red Hat AI Innovation Team ([Nayak et al., 2025](https://arxiv.org/abs/2504.07097)),

77

via [`training_hub`](https://github.com/Red-Hat-AI-Innovation-Team/training_hub) on the

78

RHAI mini-trainer backend.

79

80

**What OSFT does.** Rather than training an adapter alongside frozen weights (LoRA/QLoRA) or

81

updating every direction of every weight matrix (standard full fine-tuning), OSFT decomposes the

82

target weight matrices and learns in a subspace **orthogonal to the directions carrying the base

83

model's existing knowledge**. The intent is to absorb new domain knowledge while leaving prior

84

capability largely undisturbed — continual learning without catastrophic forgetting, and without

85

needing the original training data as a replay set.

86

87

**The one knob that matters.** `unfreeze_rank_ratio` sets how much of each weight matrix is

88

opened for adaptation. Lower values (0.1–0.3) preserve more of the original behavior; higher

89

values (0.5–0.8) allow more adaptation. This model used **0.35** — deliberately toward the

90

preservation end, because the goal was to add telecom domain knowledge to an instruction-tuned

91

model without degrading its general instruction-following.

92

93

| Parameter | Value |

94

|---|---|

95

| `unfreeze_rank_ratio` | 0.35 |

96

| Epochs | 2 |

97

| Effective batch size | 512 |

98

| Learning rate | 2e-5, cosine schedule |

99

| Max sequence length | 4096 |

100

| Loss masking | assistant turns only for SFT rows; full-document for knowledge rows |

101

| EOS token | `<turn|>` (id 106) |

102

| Precision | FP32 master weights, BF16 compute; 8-bit Adam optimizer states |

103

| Hardware | 8× AMD MI355X (gfx950), ROCm 7.2.1 |

104

105

**Relationship to the QLoRA variant.** `OTel-2.0-LLM-31B-IT-QLoRA` trains a LoRA adapter on a

106

4-bit NF4 base and merges it down to bf16. This model instead adapts the bf16 base directly in a

107

constrained subspace. Both ship as standalone bf16 HuggingFace checkpoints with no

108

quantization or adapter dependency at inference; they differ in how the weights were reached, and

109

are best compared by evaluation on your own workload rather than by method alone.

110

111

**Precision note.** The optimizer states were stored in 8-bit (torchao `AdamW8bit`) while master

112

weights, the subspace decomposition, and gradient reduction remained FP32. In controlled ablation

113

this was loss-neutral and reduced peak memory; master-weight precision is the part that must stay

114

FP32 for the model to fit the new task.

115

116

## What Changed From OTel 1.0

117

118

119

|---|---|---:|---:|

120

| Tokens | Raw documents | ~600M | ~15B |

121

| Tokens | Training | ~1B | ~440B |

122

| Data preparation | General-purpose agentic tool calling | No | Yes |

123

| Data preparation | Knowledge / fact direct QnA | No | Yes |

124

| Data preparation | RAG | Yes | Yes |

125

| Data preparation | Abstention | Yes | Yes |

126

| Data preparation | General-purpose instruction following | No | Yes |

127

| Data preparation | Base model training set | No | Yes |

128

129

Relative to OTel 1.0, OTel 2.0 increases raw source coverage by roughly 25x and training-token volume by roughly 440x. More importantly, the data mixture is broader: OTel 1.0 focused primarily on context-grounded RAG and abstention, while OTel 2.0 adds direct telecom knowledge QnA, general-purpose instruction-following and tool-use-oriented examples, and base-model-style telecom training data. Telecommunications-specific MCP, tool-calling, and instruction-following examples are not part of the current training mixture.

130

131

## Training Data

132

133

The starting corpus contains approximately **15 billion raw tokens** provided by GSMA through Open Telco AI. The corpus includes telecom standards and technical material from seven standards development and industry organizations:

134

135

| Source family | Examples |

136

|---|---|

137

| Cellular standards | 3GPP |

138

| Telecom standards and specifications | ETSI, ITU |

139

| Industry and operator materials | GSMA |

140

| Network API specifications | CAMARA |

141

| Open RAN specifications | O-RAN |

142

| Telecom business and operations frameworks | TM Forum |

143

144

Dense technical specifications from these sources were converted into material suitable for model training. The raw corpus was processed into **over 1 trillion tokens** using **Red Hat's open-source Synthetic Data Generation Hub (SDG Hub)** on **Microsoft Azure Managed Compute**, using approximately **530 GPUs**, primarily **AMD MI300X**. From this processed pool, approximately **440 billion training tokens** were generated for OTel 2.0 post-training. Model training ran on **on-premises AMD MI355X GPUs** with **Dell Technologies infrastructure and servers**.

145

146

## Data Preparation Capabilities

147

148

OTel 2.0 includes a broader supervised and synthetic-data mixture than OTel 1.0:

149

150

| Capability | Description |

151

|---|---|

152

| RAG | Context-grounded telecom answer generation from retrieved standards and technical documents |

153

| Abstention | Training examples that teach the model to avoid answering when context is missing, irrelevant, or insufficient |

154

| Direct telecom QnA | Knowledge and factual question-answer pairs for standards, protocols, services, and network concepts |

155

| Instruction following | General-purpose instruction-following examples; the current mixture does not include telecommunications-specific instruction-following examples |

156

| Agentic tool calling | General-purpose tool-calling examples; the current mixture does not include telecommunications-specific MCP or tool-calling examples |

157

158

## Intended Use

159

160

OTel-2.0-LLM-31B-IT is intended for telecom-focused applications where domain knowledge, standards familiarity, and deployment control matter. Suitable use cases include:

161

162

- Retrieval-Augmented Generation over telecom standards, specifications, and internal technical documentation.

163

- Standards interpretation and summarization for 3GPP, ETSI, GSMA, CAMARA, ITU, O-RAN, and TM Forum materials.

164

- Product development, network configuration assistance, and engineering support tasks.

165

- Telecom-specific direct QnA where the model has been separately evaluated for the target benchmark or application.

166

- Agentic workflows where a larger system provides verified tools, retrieval, validation, and audit logging.

167

168

For high-impact operational use, the model should be deployed with retrieval, source attribution, validation checks, and human review appropriate to the system risk.

169

170

### Not a Retrieval or Reranking Model

171

172

OTel 2.0 LLM checkpoints are **generative text models**. They were **not** trained for embedding, retrieval, or reranking, and should not be used to produce vector embeddings, score query-document relevance, or reorder candidate passages. The training mixture contains no contrastive, embedding, or cross-encoder reranking objectives, and no such capability has been evaluated.

173

174

In a RAG pipeline this model is the **generation** stage only. Use purpose-built models for the other stages:

175

176

| Stage | Use |

177

|---|---|

178

| Embedding / retrieval | [OTel Embedding Collection](https://huggingface.co/collections/farbodtavakkoli/otel-embedding) |

179

| Reranking | [OTel Reranker Collection](https://huggingface.co/collections/farbodtavakkoli/otel-reranker) |

180

| Generation over retrieved context | this model |

181

182

Prompting a generative model to emit relevance scores is not a substitute for a trained reranker; scores produced that way are unvalidated and should not be treated as calibrated.

183

184

---

185

186

# Serving with vLLM

187

188

This section is the deployment recipe used for this checkpoint. Follow it as written —

189

several of the settings are requirements rather than preferences, and each one is

190

explained below.

191

192

## 0. Quick start

193

194

If you just want a working server, this is the whole thing. Sections 1–8 explain each

195

choice and cover the failure modes.

196

197

```bash

198

# 1. download

199

huggingface-cli download farbodtavakkoli/OTel-2.0-LLM-31B-IT \

200

--local-dir ./OTel-2.0-LLM-31B-IT

201

202

# 2. serve (NVIDIA; for AMD see §2)

203

docker run --rm --gpus all --ipc=host --shm-size 64G -p 8000:8000 \

204

-v "$PWD/OTel-2.0-LLM-31B-IT:/model:ro" \

205

vllm/vllm-openai:latest \

206

vllm serve /model \

207

--served-model-name OTel-2.0-LLM-31B-IT \

208

--tensor-parallel-size 2 \

209

--max-model-len 8192 \

210

--dtype bfloat16 \

211

--port 8000

212

213

# 3. confirm it is really this model, not the base

214

curl -sS http://localhost:8000/v1/chat/completions \

215

-H 'Content-Type: application/json' \

216

-d '{"model":"OTel-2.0-LLM-31B-IT",

217

"messages":[{"role":"user","content":"What model is this?"}],

218

"temperature":0}' | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

219

```

220

221

Step 3 should identify **OTel 2.0, trained by AT&T Chief Data Office**. If it answers

222

"Gemma" or "Google DeepMind", you are serving the base model — wrong path or a stale

223

mount. An HTTP 200 alone does not prove the deployment is correct.

224

225

Three things that are requirements, not preferences:

226

227

1. **On AMD gfx950, set `VLLM_ROCM_USE_AITER=0`** — the default path can return malformed

228

text while still answering HTTP 200 (§2).

229

2. **No system prompt is needed** for correct self-identification; if you add one, see

230

§4 for how it must be merged.

231

3. **Use the chat endpoint**, or replicate its template exactly — `/v1/completions`

232

needs `"add_special_tokens": false` to avoid a double `<bos>` (§4, §8).

233

234

## 1. Requirements

235

236

Gemma 4 is a recent architecture. Your runtime must be new enough to contain it:

237

238

| Component | Minimum | Used for this recipe |

239

|---|---|---|

240

| vLLM | **≥ 0.19.0** | 0.23.1 |

241

| transformers | **≥ 5.5** | 5.14.0 |

242

| PyTorch | — | 2.11.0 |

243

244

An older stack fails at load with:

245

246

```

247

ValueError: The checkpoint you are trying to load has model type `gemma4_text`

248

but Transformers does not recognize this architecture.

249

```

250

251

Confirm support **before** pulling a multi-gigabyte image:

252

253

```bash

254

python3 -c "from vllm.model_executor.models.registry import ModelRegistry as R; \

255

print('Gemma4ForConditionalGeneration' in R.get_supported_archs())"

256

```

257

258

That is the architecture this repository ships. `Gemma4ForCausalLM` is the text-only

259

Gemma 4 class — a runtime that registers only that one cannot serve this checkpoint.

260

261

Note that an image tagged `latest`, `main`, or `nightly` is **not** a guarantee of a

262

recent vLLM build — check the registry rather than the tag.

263

264

## 2. Environment

265

266

### AMD (ROCm)

267

268

The container route used for this checkpoint:

269

270

```bash

271

docker run --rm --name otel_vllm \

272

--device /dev/kfd --device /dev/dri \

273

--group-add "$(getent group video | cut -d: -f3)" \

274

--ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \

275

--shm-size 64G -p 8000:8000 \

276

-e VLLM_ROCM_USE_AITER=0 \

277

-v /path/to/OTel-2.0-LLM-31B-IT:/model:ro \

278

rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 \

279

vllm serve /model \

280

--served-model-name OTel-2.0-LLM-31B-IT \

281

--tensor-parallel-size 1 \

282

--max-model-len 8192 \

283

--dtype bfloat16 \

284

--port 8000

285

```

286

287

> **`VLLM_ROCM_USE_AITER=0` is required on gfx950.** AITER is the default on that

288

> architecture and has a GEMM path that can return malformed text while the server still

289

> answers HTTP 200 — a failure that does not surface as an error. Set it to `0`.

290

> `VLLM_ROCM_USE_AITER_MOE=0` is not a substitute; this model is dense.

291

292

### NVIDIA (CUDA)

293

294

Only the container flags differ — the model, template, prompting and sampling settings

295

below are identical on both vendors:

296

297

```bash

298

docker run --rm --gpus all --ipc=host --shm-size 64G -p 8000:8000 \

299

-v /path/to/OTel-2.0-LLM-31B-IT:/model:ro \

300

vllm/vllm-openai:latest \

301

vllm serve /model \

302

--served-model-name OTel-2.0-LLM-31B-IT \

303

--tensor-parallel-size 2 \

304

--max-model-len 8192 \

305

--dtype bfloat16 \

306

--port 8000

307

```

308

309

`vllm/vllm-openai:latest` ships vLLM 0.28.0 and registers both `Gemma4ForCausalLM` and

310

`Gemma4ForConditionalGeneration`, so it satisfies the §1 requirement.

311

312

Differences from the ROCm invocation, and nothing else:

313

314

| | ROCm | CUDA |

315

|---|---|---|

316

| device flags | `--device /dev/kfd --device /dev/dri --group-add video` | `--gpus all` |

317

| device selection | `HIP_VISIBLE_DEVICES` | `CUDA_VISIBLE_DEVICES` |

318

| `VLLM_ROCM_USE_AITER=0` | **required** | omit — ROCm-only |

319

| `--ipc=host`, `--shm-size` | keep | keep (needed for tensor parallelism) |

320

321

`--tensor-parallel-size` is a sizing choice, not a vendor one; see §3.

322

323

## 3. Sizing

324

325

Weights occupy roughly **62 GB** in `bfloat16`, before KV cache and activation workspace.

326

327

| GPU | Suggested start |

328

|---|---|

329

| 1× 192 GB / 288 GB (MI300X, MI355X) | `--tensor-parallel-size 1` |

330

| 2× 80 GB (A100, H100) | `--tensor-parallel-size 2` |

331

| 1× 80 GB | `--tensor-parallel-size 1`, reduce `--max-model-len` and `--max-num-seqs`; concurrency will be limited |

332

333

`--max-model-len` must cover prompt **plus** generated tokens. Raise it only as far as

334

your workload needs — a larger value reserves KV cache that could otherwise serve

335

concurrent requests. Tune throughput with `--max-num-seqs`, and use

336

`--gpu-memory-utilization` (default `0.9`) if the GPU is shared.

337

338

## 4. Prompting

339

340

### No system prompt is required

341

342

The model knows its own identity from the user turn alone. Measured against this

343

checkpoint at `temperature: 0`, with **no system message sent**:

344

345

| asked | answered |

346

|---|---|

347

| *"What model is this?"* | *"I'm OTel 2.0, a large language model trained by AT&T Chief Data Office."* |

348

| *"Who made you?"* | *"I'm OTel 2.0, trained by AT&T Chief Data Office."* |

349

| *"Are you Gemma?"* | *"I'm OTel 2.0… My base model was Gemma 4 31B-IT, but the telecom post-training that makes me OTel 2.0 was done by AT&T Chief Data Office."* |

350

351

Identity is trained into the weights directly rather than supplied at inference time, so

352

there is no identity prompt to forget.

353

354

### Adding your own system prompt

355

356

Pass it as the first message in the normal way — it is used as-is, with no special

357

handling and nothing to merge:

358

359

```python

360

def build_messages(user_content, system_prompt=None):

361

msgs = []

362

if system_prompt:

363

msgs.append({"role": "system", "content": system_prompt})

364

msgs.append({"role": "user", "content": user_content})

365

return msgs

366

367

build_messages("Summarize the purpose of the 3GPP SA5 working group.")

368

369

build_messages("Determine the efficiency of the derived binary sequence ...",

370

system_prompt="You are an expert problem solver. Solve numerical "

371

"exercises step by step and give the final answer "

372

"in \\boxed{}.")

373

```

374

375

This matches how the model was trained: reasoning examples carry a **task-only** system

376

turn ("You are an expert problem solver…"), and every other category trains with no system

377

turn at all.

378

379

### Chat template

380

381

The repository ships the correct `chat_template.jinja`, and vLLM applies it

382

automatically. The model was trained to begin its answer **directly** after the model

383

turn:

384

385

```

386

<|turn>model\n

387

```

388

389

Do not substitute the upstream Gemma 4 template. Some Gemma 4 templates open a *thought*

390

channel on the generation prompt (`<|turn>model\n<|channel>thought\n<channel|>`); this

391

model was not trained to produce or consume that channel, and using such a template

392

degrades output formatting on every request. If you supply your own template with

393

`--chat-template`, verify the rendered generation prompt ends at `<|turn>model\n`.

394

395

### Stop tokens

396

397

The model emits `<turn|>` (id `106`) to end a turn; id `1` is the standard EOS. Both are

398

declared in `generation_config.json` and applied by vLLM automatically. Only set

399

`stop_token_ids` explicitly if you are bypassing the chat endpoint — omitting id `106`

400

lets generations run to the token limit.

401

402

### If you use `/v1/completions` instead of `/v1/chat/completions`

403

404

Build the prompt with `apply_chat_template(messages, add_generation_prompt=True)` and trim

405

it to end exactly at `<|turn>model\n`. Then set **`"add_special_tokens": false`** — the

406

chat template already prepends `<bos>`, and a second one corrupts the prompt — and pass

407

`"stop_token_ids": [1, 106]` explicitly, since the completions endpoint does not apply the

408

chat template's stop tokens for you.

409

410

## 5. Sampling parameters

411

412

For deterministic, reproducible output — recommended for evaluation, RAG, extraction, and

413

anything requiring a fixed answer format:

414

415

```json

416

{

417

"temperature": 0,

418

"top_p": 1.0,

419

"max_tokens": 512

420

}

421

```

422

423

`temperature: 0` selects greedy decoding, which is what the deterministic recipe uses.

424

For open-ended drafting or summarization, `temperature` in the `0.2`–`0.7` range with

425

`top_p` around `0.9`–`0.95` is a reasonable starting point.

426

427

Size `max_tokens` to the task. Short-form and multiple-choice answers finish well inside

428

512 tokens; step-by-step reasoning or long structured output should be given

429

approximately 1280. An over-large value costs nothing when the model stops on its own,

430

but an under-sized one truncates answers mid-format.

431

432

## 6. Client examples

433

434

### curl

435

436

```bash

437

curl -sS http://localhost:8000/v1/chat/completions \

438

-H "Content-Type: application/json" \

439

-d '{

440

"model": "OTel-2.0-LLM-31B-IT",

441

"messages": [

442

{"role": "user", "content": "Summarize the purpose of the 3GPP SA5 working group."}

443

],

444

"temperature": 0,

445

"top_p": 1.0,

446

"max_tokens": 512

447

}'

448

```

449

450

Raw-prompt equivalent via `/v1/completions` (note `add_special_tokens: false`):

451

452

```bash

453

curl -sS http://localhost:8000/v1/completions \

454

-H "Content-Type: application/json" \

455

-d '{

456

"model": "OTel-2.0-LLM-31B-IT",

457

"prompt": "<bos><|turn>user\nWhat model are you?<turn|>\n<|turn>model\n",

458

"max_tokens": 256,

459

"temperature": 0,

460

"add_special_tokens": false,

461

"stop_token_ids": [1, 106]

462

}'

463

```

464

465

### OpenAI Python SDK

466

467

```python

468

from openai import OpenAI

469

470

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

471

472

resp = client.chat.completions.create(

473

model="OTel-2.0-LLM-31B-IT",

474

messages=build_messages("Summarize the purpose of the 3GPP SA5 working group."),

475

temperature=0,

476

top_p=1.0,

477

max_tokens=512,

478

)

479

print(resp.choices[0].message.content)

480

```

481

482

`build_messages` is the small helper from [§4](#4-prompting); it simply adds your system

483

prompt as the first message when you have one.

484

485

### Offline batch inference

486

487

```python

488

from vllm import LLM, SamplingParams

489

490

llm = LLM(model="/path/to/OTel-2.0-LLM-31B-IT",

491

dtype="bfloat16", max_model_len=8192, tensor_parallel_size=1)

492

493

params = SamplingParams(temperature=0, top_p=1.0, max_tokens=512)

494

out = llm.chat([build_messages("...")], params)

495

print(out[0].outputs[0].text)

496

```

497

498

Use `llm.chat(...)` rather than `llm.generate(...)` with raw strings, so the repository's

499

chat template is applied. If you must pre-tokenize, apply the template yourself with

500

`add_generation_prompt=True` and pass token IDs via `TokensPrompt`.

501

502

### Sending an image

503

504

The checkpoint is multimodal, so the standard OpenAI image content type works with no extra

505

server flags — vLLM sets up its multimodal encoder cache from `config.json` at startup.

506

507

```python

508

import base64, json, urllib.request

509

510

img = base64.b64encode(open("diagram.png", "rb").read()).decode()

511

512

body = json.dumps({

513

"model": "OTel-2.0-LLM-31B-IT",

514

"temperature": 0,

515

"max_tokens": 256,

516

"messages": [{"role": "user", "content": [

517

{"type": "image_url",

518

"image_url": {"url": f"data:image/png;base64,{img}"}},

519

{"type": "text", "text": "Describe what this diagram shows."},

520

]}],

521

}).encode()

522

523

req = urllib.request.Request("http://localhost:8000/v1/chat/completions",

524

data=body, headers={"Content-Type": "application/json"})

525

with urllib.request.urlopen(req, timeout=240) as r:

526

print(json.load(r)["choices"][0]["message"]["content"])

527

```

528

529

Images consume the same context budget as text, so `--max-model-len` must accommodate the

530

encoded image tokens plus your prompt and the generation. Read

531

[Vision and Modality](#vision-and-modality) before relying on image output: the vision

532

encoder is the untuned base model's and has not been benchmarked for this release.

533

534

## 7. Verifying a deployment

535

536

An HTTP 200 does not by itself prove the deployment is correct. After starting the

537

server, check:

538

539

```bash

540

curl -sS http://localhost:8000/v1/models

541

```

542

543

Then send `What model is this?` with **no system prompt** — the response should identify

544

OTel 2.0 trained by AT&T Chief Data Office. If it describes itself as Gemma or a Google

545

DeepMind model, you are not serving the weights you think you are (a base-model checkpoint,

546

or a stale mount).

547

548

Check for stray `<|channel>` or `thought` markup in responses; its presence means a

549

non-matching chat template is in use (§4).

550

551

For a substantive check, run a set of prompts representative of your workload at

552

`temperature: 0` and compare against the same prompts run through `transformers`. Compare

553

**the same prompts**, not aggregate scores from different sample sets — comparing a small

554

sample against a full-set average produces large, meaningless differences.

555

556

## 8. Troubleshooting

557

558

| Symptom | Cause | Resolution |

559

|---|---|---|

560

| `ValueError: ... model type gemma4_text ... not recognize` | vLLM/transformers too old | vLLM ≥ 0.19, transformers ≥ 5.5 (§1) |

561

| `Attempted to load weight (512) into parameter (256)` | vLLM build lacks the Gemma 4 loader — Gemma 4 uses a heterogeneous per-layer attention layout | upgrade to a vLLM release listing Gemma 4 support (§1) |

562

| Fluent but malformed or nonsensical text, HTTP 200, on gfx950 | AITER GEMM path | `VLLM_ROCM_USE_AITER=0` (§2) |

563

| Model says it is Gemma / Google DeepMind | not the fine-tuned weights — wrong path or stale mount | verify the served directory (§7) |

564

| Stray thought/channel markup in responses | Non-matching chat template | Use the repository's `chat_template.jinja` (§4) |

565

| Garbled output when using `/v1/completions` | double `<bos>` — template already prepends one | set `"add_special_tokens": false` (§4) |

566

| Generations run to `max_tokens` and get truncated | Stop token `106` not applied | Use the chat endpoint, or include id `106` in `stop_token_ids` (§4) |

567

| OOM at startup | KV cache does not fit | Lower `--max-model-len` / `--max-num-seqs`, or raise `--tensor-parallel-size` (§3) |

568

| Hub shows `Configuration Parsing Warning: In config.json: "num_experts" must be a number` | Hub-side schema notice, **not** a defect in the weights | Nothing to do — see below |

569

570

---

571

572

## Limitations and Responsible Use

573

574

### Supported Language and Modality

575

576

OTel 2.0 targets **English-language** use. The checkpoint is architecturally multimodal and

577

accepts image input, but **all training and all quality claims in this card are text-only**.

578

Audio and video are not supported. See [Vision and Modality](#vision-and-modality).

579

580

### Vision and Modality

581

582

**This checkpoint is multimodal, but only its text path was trained.** That combination is

583

deliberate and worth stating precisely, because "accepts images" and "was trained on images"

584

are different claims and only the first is true here.

585

586

The base model, `gemma-4-31b-it`, is natively multimodal

587

(`Gemma4ForConditionalGeneration`): a `vision_tower`, an `embed_vision` projection into the

588

language model's representation space, and the text decoder. OSFT post-training targeted

589

**only** the language path — the attention and MLP projections under

590

`model.language_model.layers.*`. No image was present in the training corpus, so no image

591

gradient ever existed, and the vision weights were carried through unchanged.

592

593

Verified against the base model tensor by tensor:

594

595

596

|---|---|---|---|

597

| `vision_tower` | 355 | **bit-identical** | 0.000e+00 |

598

| `embed_vision` (projector) | 1 | **bit-identical** | 0.000e+00 |

599

| language model | 832 | trained | up to 4.9e-02 |

600

601

So the vision encoder in this repository is **stock Gemma 4, with no telecom adaptation**.

602

Image inputs are handled by the base model's visual understanding; the OTel training shows

603

up only after the projection, in the language model that reasons over those features.

604

605

**Text quality is unaffected by shipping the vision tower — measured, not assumed.** The

606

full 1,700-row eval was run through both builds on real served vLLM endpoints under

607

identical settings:

608

609

| | MICRO | MACRO |

610

|---|---:|---:|

611

| text-only build | 1567/1700 (92.18%) | 94.83% |

612

| this multimodal build | 1563/1700 (91.94%) | 94.65% |

613

614

Difference: −0.19pp MACRO, with 9 rows correct only in the text-only build and 5 only in the

615

multimodal one — McNemar exact two-sided **p = 0.42**, i.e. ordinary run-to-run

616

nondeterminism rather than a systematic effect.

617

618

**The image path is functional.** It was exercised end to end on the served endpoint: the

619

model correctly described a synthetic geometric image, and correctly read four values off a

620

rendered telecom readout (PCI, SINR in dB, throughput in Mbps, band). The path works.

621

622

> **What is still not established.** Those are functional smoke tests, not a vision

623

> benchmark. The vision tower has had **no telecom-specific training**, and this release has

624

> not been evaluated on any image benchmark — general or domain. Image-based use of network

625

> diagrams, spectrum plots, equipment photographs, or scanned specifications requires its own

626

> task-specific evaluation before deployment. Every accuracy number elsewhere in this card is

627

> text-only.

628

629

Serving is unchanged: same flags, same prompt format, same sampling as a text-only build.

630

vLLM selects the architecture from `config.json` and initializes its multimodal encoder cache

631

automatically. See [§6 Client examples](#6-client-examples) for sending an image.

632

633

### Tool Use and Instruction Following

634

635

The training data includes **general-purpose tool-calling and instruction-following examples**, but it does not currently include:

636

637

- Telecommunications-specific MCP examples.

638

- Telecommunications-specific tool-calling examples.

639

- Telecommunications-specific instruction-following examples.

640

641

General-purpose tool-use training should not be interpreted as readiness to operate telecommunications tools autonomously. This includes multi-step tasks where a scorer measures the resulting network state rather than the text of a response; general-purpose tool-use training does not establish competence at such closed-loop operational workflows (see [Autonomous Operational Task Completion Not Benchmarked](#autonomous-operational-task-completion-not-benchmarked)). Agentic deployment requires an external tool runtime, validated tool schemas, permission controls, audit logging, safeguards, and human review appropriate to the risk. The model alone does not guarantee correct or safe tool execution.

642

643

### Telecommunications Data Not Included in Training

644

645

The current training mixture does not include the following classes of operational telecommunications data:

646

647

- **Event data:** Elasticsearch records covering user activity, anomalies, failures, IMS events, and RADIUS authentication.

648

- **Network-performance data:** KPIs, 5G performance metrics, and Passive Intermodulation (PIM) interference data.

649

- **RF and spectral data:** Field-test results, antenna-port metrics, signal diagnostics, heatmaps, and noise or interference measurements.

650

- **5G core data:** Control-plane and inter-Network Function (NF) signaling data.

651

- **IETF RFCs:** The training corpus draws on 3GPP, ETSI, ITU, GSMA, CAMARA, O-RAN, and TM Forum material; it does not include a dedicated corpus of IETF RFCs (e.g., routing, MPLS, BGP, or other protocol RFCs).

652

- **Vendor CLI and network operating system (NOS) documentation:** Command references and configuration guides for vendor platforms such as Cisco IOS-XR and other Cisco NOS variants, Juniper JUNOS, DNOS, Nokia BNG platforms, Arista EOS, and similar equipment are not part of the training mixture.

653

- **Operator-specific network design and configuration artifacts:** Internal network design documents, control-plane design references, customer/equipment configuration records, and similar operator-proprietary engineering artifacts are not included.

654

- **OSS/BSS and incident/change-management workflow tooling:** Fault- and event-management consoles, IT service-management ticketing, and change-approval systems (e.g., alarm dashboards, trouble-ticket lifecycles, and MOP approval and closure records). Standards familiarity with TM Forum process frameworks does not imply validated ability to execute an incident through these operational systems.

655

656

Standards knowledge should not be interpreted as experience with live operator telemetry, private network records, vendor-specific CLI/NOS behavior, or these excluded operational datasets. Applications involving these data types require separate task-specific evaluation, grounding, and validation.

657

658

### Operational Procedure (MOP) Validation Not Performed

659

660

The model has not been validated against **Methods of Procedure (MOPs)** — step-by-step operational runbooks (e.g., maintenance activities, configuration changes, turn-ups, failovers, troubleshooting sequences) — executed or emulated against live or digital-twin network device instances. No benchmarking has been performed to confirm that the model's step-by-step procedural reasoning (command sequencing, expected outputs, checkpoints, rollback steps) is correct or safe when applied to real or emulated network operating system instances. Standards and configuration-assistance knowledge demonstrated elsewhere in this model card should not be interpreted as validated readiness for autonomous or human-in-the-loop execution of operational procedures. Any such use requires dedicated MOP-based evaluation (e.g., against emulated NOS instances in a digital twin environment) before deployment.

661

662

### Autonomous Operational Task Completion Not Benchmarked

663

664

The model has not been evaluated for autonomous, closed-loop completion of operational telecom tasks whose success is measured by the resulting state of a live or emulated network, rather than by the text of a response. Task families of this kind — extracting quantitative SLA/QoS parameters from a natural-language service request; bringing up and configuring a private 5G standalone core and RAN to meet measured latency and throughput targets; diagnosing and repairing a live routing fault through an alarm-to-closure workflow; and standing up a network defense that must withstand adversarial traffic — exercise capabilities the current training mixture targets but has not validated. Demonstrated standards knowledge and configuration assistance should not be read as readiness for autonomous completion of these tasks, which requires task-specific evaluation with outcome-based scoring before any non-supervised use.

665

666

### Network Security Operations Not Validated

667

668

The model has not been trained or evaluated for defensive network-security operations — traffic and anomaly analysis, firewall/packet-filter or eBPF program construction, DDoS detection and mitigation, or deployment of long-running protective processes on network appliances. Security-operations use requires dedicated evaluation and human oversight.

669

670

### General Reliability and Deployment Limitations

671

672

- This is the **OSFT** variant: the base weights were adapted in place within a rank-limited orthogonal subspace (`unfreeze_rank_ratio=0.35`), not through an adapter. Where training cost matters more, compare against the QLoRA variant `OTel-2.0-LLM-31B-IT-QLoRA`. See [OSFT Training Details](#osft-training-details).

673

- OTel 2.0 is domain-specific to telecommunications and should not be treated as a general-purpose model for unrelated fields.

674

- Telecom standards evolve over time; answers should be checked against the relevant document version and release.

675

- RAG quality depends on document ingestion, chunking, retrieval, reranking, prompt design, and source freshness.

676

- Direct QnA behavior should be evaluated separately from RAG behavior; strong performance in one setting does not imply strong performance in the other.

677

- For high-impact operational use, deploy the model with retrieval, source attribution, validation checks, and human review appropriate to the system risk.

678

679

## Future Work

680

681

The following work is planned and should not be interpreted as a capability claim for the current checkpoint:

682

683

- Release the exact OTel 2.0 training implementation and reproducible configuration.

684

- Release a comprehensive OTel 2.0 evaluation through MLPeFT in collaboration with MLCommons.

685

- Expand independent evaluation across direct question answering, RAG, abstention, instruction following, safety, and operational telecom tasks.

686

- Address the documented gaps in operational data, telecom-specific tool use, multilingual support, and telecom-specific multimodal evaluation.

687

- Evaluate Methods of Procedure, digital-twin workflows, closed-loop network tasks, and security operations before making operational-readiness claims.

688

- Work with the AWS and Tenstorrent teams to add and verify model training and inference support on their hardware platforms.

689

- Compare compatible training stacks for GRPO and multi-node training under a common benchmarking methodology.

690

- Benchmark AMD Ryzen inference across multiple models, inference stacks, and quantization formats.

691

692

## Related Models

693

694

**Other builds of this same model:**

695

696

| Variant | How it was trained | When to prefer it |

697

|---|---|---|

698

| `OTel-2.0-LLM-31B-IT` (this repo) | OSFT — orthogonal subspace, `unfreeze_rank_ratio=0.35` | Default choice |

699

| [`OTel-2.0-LLM-31B-IT-QLoRA`](https://huggingface.co/farbodtavakkoli/OTel-2.0-LLM-31B-IT-QLoRA) | LoRA adapter (`r=256`, `alpha=512`) on a 4-bit NF4 base, merged to bf16 | Cheaper to reproduce or iterate on |

700

701

Both ship as standalone bf16 checkpoints needing no adapter or quantization library at

702

inference, and both use the same prompting, chat template, stop tokens, and sampling guidance

703

documented above. They differ in how the weights were reached; compare them by evaluation on

704

your own workload rather than by method alone.

705

706

**Collections:**

707

708

- [OTel LLM Collection](https://huggingface.co/collections/farbodtavakkoli/otel-llm)

709

- [OTel Embedding Collection](https://huggingface.co/collections/farbodtavakkoli/otel-embedding)

710

- [OTel Reranker Collection](https://huggingface.co/collections/farbodtavakkoli/otel-reranker)

711

712

## Project Resources

713

714

- Project page: https://huggingface.co/farbodtavakkoli

715

- Code: https://github.com/farbodtavakkoli/OTel

716

- OTel 1.0 media coverage list: https://github.com/farbodtavakkoli/OTel/blob/main/docs/OTel-1.0-media-coverage.md

717

- OTel 2.0 blogs: https://github.com/farbodtavakkoli/OTel/blob/main/docs/OTel-2.0-blogs.md

718

719

## Contributors and Organizations

720

721

Contributors to the OTel 2.0 release and supporting infrastructure include:

722

723

| Organization | Contributors |

724

|---|---|

725

| AT&T | Farbod Tavakkoli, Jorden Terrazas, Roderic Paulk, Sharath Japa, Tzvi Chumash, Miguel Armenta, Pavan Tagirisa, Kostikey Mustakas, Mark Austin, Andy Markus |

726

| MLCommons | Gregory Diamos, David Kanter, Kenneth Church |

727

| Microsoft | Chunyu Li, Gulsimo Osimi, Rick Lievano, Inayat Wali, Manoj Bableshwar, Marie-Louise Onga Nana, Naomi Moneypenny, Osi Otugo, Rahul Kumar, SeokJin Han, Steve Sweetman, Trinidad Salazar, Ven Kumar, Vivek Ramaswamy |

728

| AMD | Alexander Finn, Andy Allred, Andrey Ivannikov, Antti-Ville Suni, Mark van Heeswijk, Kumaran Siva, Curt Wortman, Mehrvash Poole, Eric Lynn |

729

| Dell | Brian Sweeney, Suzanne Randall, Patrick Allen, Matt Currie, Justin Wilson, Jason Kane, Keith Napoleon, Sarah Lake, Mike Hess, Randy Tornes |

730

| GSMA | Louis Powell, Zeinab Nezami, Enrique Molero |

731

| Red Hat | Aditi Saluja, William Caban, Shivchander Sudalairaj, Kai Xu, Ravi Sharma, Hanen Garcia, Joe Crispo, Eshwar Sivaramakrishnan |

732

| Pleias | Anastasia Stasenko, Pierre-Carl Langlais, Mohamed Chenene, Carlos Rosas, Yannick Detrois |

733

734

Organizations involved in the OTel 2.0 release and supporting infrastructure include AT&T, MLCommons, Microsoft, AMD, Dell, GSMA, Red Hat, and Pleias.

735

736

## Citation

737

738

```bibtex

739

@misc{otel_2_models_2026,

740

title = {OTel 2.0: Open Telco AI Datasets, Benchmarks, and Models},

741

author = {Tavakkoli, Farbod and others},

742

year = {2026},

743

note = {Open Telco (OTel 2.0) model release},

744

url = {https://huggingface.co/farbodtavakkoli},

745

organization = {AT\&T, MLCommons, Microsoft, AMD, Dell, GSMA, Red Hat, Pleias}

746

}

747

```

748

749

## Contact

750

751

For technical questions, contact farbod.tavakkoli@att.com or farbodtavakoli@gmail.com.

752