README.md
32.7 KB · 609 lines · markdown Raw
1 ---
2 license: other
3 license_name: exaone
4 license_link: LICENSE
5 pipeline_tag: tabular-classification
6 tags:
7 - tabular
8 - tabular-classification
9 - tabular-regression
10 - in-context-learning
11 - foundation-model
12 - pytorch
13 - safetensors
14 - exaone
15 metrics:
16 - accuracy
17 ---
18
19 <br>
20
21 <div align="center">
22 <img src="assets/exaone_logo.png" alt="EXAONE Tabular" width="160">
23 <h1>EXAONE Tabular</h1>
24 </div>
25
26 <br>
27
28 <div align="center">
29 <a href="https://huggingface.co/LG-AI-Research/EXAONE-Tabular" style="text-decoration: none;">
30 <img src="https://img.shields.io/badge/🤗-HuggingFace-FC926C?style=for-the-badge" alt="HuggingFace">
31 </a>
32 <a href="https://github.com/LGAI-Research/EXAONE-Tabular" style="text-decoration: none;">
33 <img src="https://img.shields.io/badge/🖥️-GitHub-2B3137?style=for-the-badge" alt="GitHub">
34 </a>
35 </div>
36
37 <br><br>
38
39 **EXAONE Tabular** is a transformer-based **foundation model for tabular data** that solves
40 **classification** and **regression** through **in-context learning**: you pass the labeled
41 rows to `fit` and the model predicts new rows in a single forward pass — **no gradient
42 updates and no per-dataset training**.
43
44 This repository is the **`exaonetabular` inference runtime** — a self-contained package
45 that loads a released checkpoint and serves predictions through a small, scikit-learn-style API.
46 The code here is permissively licensed; the released **weights are non-commercial** — see
47 [License](#license).
48
49 Both checkpoints are released: `EXAONETabularClassifier` and `EXAONETabularRegressor` each fetch
50 their own weights with a single `from_pretrained()` call. See
51 [Available checkpoints](#available-checkpoints).
52
53 For more details, please refer to the [GitHub repository](https://github.com/LGAI-Research/EXAONE-Tabular) and our [technical report](https://huggingface.co/LG-AI-Research/EXAONE-Tabular/blob/main/EXAONE_Tabular_v1.0_Technical_Report.pdf).
54 ## Model Configuration
55
56 <div style="background-color: rgba(128, 128, 128, 0.1); border-radius: 12px; padding: 12px 24px;">
57
58 - Model Type: In-context tabular foundation model (Cross-axis Summary Transformer (CAST))
59
60 - Embedding dimension: 192
61 - Attention heads: 6
62 - Transformer layers: 12
63 - Feed-forward expansion: 4x
64 - MLP sharing: Single
65 - Feature-attention operations per layer: 2
66 - Feature-level summary tokens: 3
67 - Row-level summary tokens: 32
68 - Attention normalization: SSMax
69 - Total parameters
70 - Classification: 20,807,866 (≈20.8M)
71 - Regression: 21,110,247 (≈21.1M)
72
73 </div>
74
75
76 ## Evaluation Results
77
78 ### TabArena
79
80 EXAONE Tabular achieves an overall Elo of **1,755** on TabArena without per-dataset tuning or ensembling. It ranks second overall, first on classification with an Elo of **1,759**, and second on regression with an Elo of **1,883**, while using approximately 21M parameters.
81
82 All TabArena results reported below are taken from the official [TabArena leaderboard](https://huggingface.co/spaces/TabArena/leaderboard).
83
84 <div align="center">
85 <img src="figures/tabarena_overall_elo_model_parameters.svg" alt="Overall TabArena Elo versus model parameters" width="92%">
86 <br>
87 <em>Figure 1. Overall TabArena Elo versus model size. EXAONE Tabular achieves competitive performance with approximately 21M parameters.</em>
88 </div>
89
90 <div align="center">
91 <img src="figures/tabarena_elo_cls_reg.svg" alt="TabArena classification Elo on the left and regression Elo on the right" width="94%">
92 <br>
93 <em>Figure 2. TabArena Elo scores on classification (left) and regression (right) tasks.</em>
94 </div>
95
96 **Table 1.** *TabArena Elo scores and computational costs. Training and prediction costs are median seconds per 1,000 samples.*
97
98 <table>
99 <thead>
100 <tr>
101 <th bgcolor="#EEF2FF" align="left">Model type</th>
102 <th bgcolor="#EEF2FF" align="left">Model</th>
103 <th bgcolor="#EEF2FF" align="left">Configuration</th>
104 <th bgcolor="#EEF2FF" align="right">Overall Elo ↑</th>
105 <th bgcolor="#EEF2FF" align="right">Cls. Elo ↑</th>
106 <th bgcolor="#EEF2FF" align="right">Reg. Elo ↑</th>
107 <th bgcolor="#EEF2FF" align="right">Cost: train / predict (s/1K) ↓</th>
108 </tr>
109 </thead>
110 <tbody>
111 <tr bgcolor="#F5F8FF" style="color:#1E3A8A; font-weight:700;"><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>Foundation Model</strong></font></td><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>EXAONE Tabular</strong></font></td><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>Default</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,755</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,759</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>1,883</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>5.79 / 0.605</strong></font></td></tr>
112 <tr><td>Foundation Model</td><td>TabFM</td><td>Default</td><td align="right">1,765</td><td align="right">1,746</td><td align="right">1,993</td><td align="right">38.81 / 6.985</td></tr>
113 <tr><td>Foundation Model</td><td>TabPFN-3</td><td>Default</td><td align="right">1,642</td><td align="right">1,635</td><td align="right">1,793</td><td align="right">3.66 / 0.399</td></tr>
114 <tr><td>Foundation Model</td><td>TabPFN-2.6</td><td>Default</td><td align="right">1,592</td><td align="right">1,586</td><td align="right">1,734</td><td align="right">5.48 / 0.555</td></tr>
115 <tr><td>Foundation Model</td><td>RealTabPFN-2.5</td><td>Tuned + ensembled</td><td align="right">1,572</td><td align="right">1,562</td><td align="right">1,731</td><td align="right">2,040.22 / 8.908</td></tr>
116 <tr><td>Foundation Model</td><td>TabICLv2</td><td>Default</td><td align="right">1,569</td><td align="right">1,574</td><td align="right">1,672</td><td align="right">2.05 / 0.151</td></tr>
117 <tr><td>Neural Network</td><td>RealMLP</td><td>Tuned + ensembled</td><td align="right">1,482</td><td align="right">1,468</td><td align="right">1,648</td><td align="right">2,950.72 / 11.975</td></tr>
118 <tr><td>Foundation Model</td><td>TabDPT</td><td>Tuned + ensembled</td><td align="right">1,437</td><td align="right">1,398</td><td align="right">1,715</td><td align="right">4,910.38 / 286.537</td></tr>
119 <tr><td>Neural Network</td><td>TabM</td><td>Tuned + ensembled</td><td align="right">1,426</td><td align="right">1,446</td><td align="right">1,449</td><td align="right">2,450.13 / 2.247</td></tr>
120 <tr><td>Tree-based</td><td>LightGBM</td><td>Tuned + ensembled</td><td align="right">1,410</td><td align="right">1,415</td><td align="right">1,477</td><td align="right">417.05 / 2.639</td></tr>
121 <tr><td>Tree-based</td><td>CatBoost</td><td>Tuned + ensembled</td><td align="right">1,398</td><td align="right">1,398</td><td align="right">1,484</td><td align="right">1,346.21 / 0.344</td></tr>
122 <tr><td>Tree-based</td><td>XGBoost</td><td>Tuned + ensembled</td><td align="right">1,357</td><td align="right">1,364</td><td align="right">1,404</td><td align="right">693.49 / 1.689</td></tr>
123 <tr><td>Foundation Model</td><td>TabSwift</td><td>Default</td><td align="right">1,334</td><td align="right">1,339</td><td align="right">1,397</td><td align="right">1.18 / 0.072</td></tr>
124 <tr><td>Foundation Model</td><td>Nori-30M</td><td>Default</td><td align="right">1,156</td><td align="right">—</td><td align="right">1,753</td><td align="right">0.53 / 0.080</td></tr>
125 </tbody>
126 </table>
127
128 Elo scores are reported for the same configuration shown in the `Configuration` column. Cost is the median training and prediction time per 1,000 samples from the overall TabArena results; lower is better. Nori-30M has no classification result in the source data.
129
130 ### ScoringBench
131
132 The results below are taken from the official [ScoringBench leaderboard](https://scoringbench.com/). The reported average ranks were computed against the full ScoringBench comparison pool of 51 model entries across 102 datasets. Figure 3 visualizes a selected set of 28 models, while Table 2 displays only the 10 model families that overlap with the TabArena table; neither the figure nor the table recomputes ranks on these subsets.
133
134 <div align="center">
135 <img src="figures/scoringbench_r2_crps_site_style_tikz.svg" alt="ScoringBench R-squared and CRPS mean ranks" width="92%">
136 <br>
137 <em>Figure 3. Mean-rank comparison on ScoringBench across R² and CRPS; lower ranks are better.</em>
138 </div>
139
140 **Table 2.** *ScoringBench average ranks for models also included in the TabArena comparison. Lower is better.*
141
142 <table>
143 <thead>
144 <tr>
145 <th bgcolor="#EEF2FF" align="left">Model</th>
146 <th bgcolor="#EEF2FF" align="right">R² average rank ↓</th>
147 <th bgcolor="#EEF2FF" align="right">CRPS average rank ↓</th>
148 </tr>
149 </thead>
150 <tbody>
151 <tr bgcolor="#F5F8FF" style="color:#1E3A8A; font-weight:700;"><td style="color:#1E3A8A;"><font color="#1E3A8A"><strong>EXAONE Tabular</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>8.75</strong></font></td><td align="right" style="color:#1E3A8A;"><font color="#1E3A8A"><strong>6.07</strong></font></td></tr>
152 <tr><td>TabPFN-3</td><td align="right">10.03</td><td align="right">6.16</td></tr>
153 <tr><td>Nori-30M</td><td align="right">14.90</td><td align="right">11.59</td></tr>
154 <tr><td>TabICLv2</td><td align="right">15.99</td><td align="right">11.88</td></tr>
155 <tr><td>TabPFN-2.6</td><td align="right">16.87</td><td align="right">18.89</td></tr>
156 <tr><td>RealTabPFN-2.5</td><td align="right">20.70</td><td align="right">19.96</td></tr>
157 <tr><td>CatBoost</td><td align="right">33.52</td><td align="right">34.62</td></tr>
158 <tr><td>TabM</td><td align="right">35.95</td><td align="right">33.79</td></tr>
159 <tr><td>RealMLP</td><td align="right">36.14</td><td align="right">34.05</td></tr>
160 <tr><td>XGBoost</td><td align="right">38.83</td><td align="right">38.76</td></tr>
161 </tbody>
162 </table>
163
164 The benchmark-specific ScoringBench entries for RealTabPFN-2.5, CatBoost, TabM, RealMLP, and XGBoost are `tabpfn_realv2_5`, `catboost_quantile`, `tabm_d`, `pytabkit_realmlp_td`, and `xgb_vector`, respectively. Models are matched across the two benchmarks by model family, so their configurations may differ: for example, Table 1 reports the tuned-and-ensembled RealTabPFN-2.5, whereas Table 2 uses the `tabpfn_realv2_5` entry evaluated by ScoringBench.
165
166
167 ## Requirements
168
169 - **Python** ≥ 3.11
170 - **PyTorch** ≥ 2.6, < 3 &nbsp;(a **CUDA GPU is strongly recommended** — the model uses fused
171 attention kernels and half precision; CPU inference works but is slow)
172 - NumPy ≥ 2.3.5 · scikit-learn ≥ 1.7.2 · safetensors ≥ 0.4 · huggingface_hub ≥ 0.24
173 &nbsp;(floors are the versions this release was validated against)
174
175 Install the package — the dependencies above come with it:
176
177 ```bash
178 pip install "exaonetabular @ git+https://github.com/LGAI-Research/EXAONE-Tabular.git"
179 ```
180
181 From a checkout, `pip install .` (add `-e` for an editable install) or `uv sync` do the same.
182
183 `huggingface_hub` is included, so `from_pretrained` can fetch the released weights out of the box.
184 Downloads honor the standard Hub environment (`HF_HOME` for the cache, `HF_TOKEN` for a gated repo).
185
186 Verify the install:
187
188 ```python
189 import exaonetabular
190 print(exaonetabular.__version__)
191 ```
192
193 > Dependency ranges are declared in
194 > [`pyproject.toml`](https://github.com/LGAI-Research/EXAONE-Tabular/blob/main/pyproject.toml)
195 > (distribution name `exaonetabular`).
196
197
198 ## Quickstart
199
200 EXAONE Tabular ships as **scikit-learn-style estimators**. `EXAONETabularClassifier` and
201 `EXAONETabularRegressor` both expose the familiar `fit` / `predict` surface, return `self` from
202 `fit`, and set the usual fitted attributes — `classes_`, `n_classes_` and `n_features_in_` on the
203 classifier, `n_features_in_` on the regressor — so they slot into the workflow you already use,
204 including as the final step of a `sklearn.pipeline.Pipeline`. `predict_proba` is classification
205 only; the regressor returns point estimates from `predict`.
206
207 `from_pretrained` handles the rest in one call: it fetches that task's released checkpoint from the
208 Hub, builds the model from its frozen manifest, and loads the weights. The repo id, revision, and
209 architecture are baked into the package for both tasks, so there is nothing to configure by hand.
210
211 Both snippets below run as written, on a stock scikit-learn dataset.
212
213 > **Inputs are NumPy arrays.** `X` is 2-D `float` (rows × features); `y` is 1-D — class labels for
214 > classification, real values for regression. Anything else raises
215 > `TypeError: features must be a NumPy array`.
216
217 > **scikit-learn interop.** These estimators implement the estimator *interface* — including
218 > `__sklearn_is_fitted__` and `__sklearn_tags__`, so `check_is_fitted`, `is_classifier` /
219 > `is_regressor`, and use as the final step of a `Pipeline` all work. They do not subclass
220 > `BaseEstimator`, so there is no `get_params` / `set_params` / `score`, and `clone`,
221 > `cross_val_score`, and `GridSearchCV` are therefore not supported.
222
223 <details open>
224 <summary><b>Classification</b></summary>
225
226 ```python
227 from sklearn.datasets import load_breast_cancer
228 from sklearn.model_selection import train_test_split
229
230 from exaonetabular import EXAONETabularClassifier
231
232 X_train, X_test, y_train, y_test = train_test_split(
233 *load_breast_cancer(return_X_y=True), test_size=0.25, random_state=0
234 )
235
236 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0") # download + verify + load
237
238 clf.fit(X_train, y_train) # no training — stores context + fits preprocessors
239 proba = clf.predict_proba(X_test) # (n_samples, n_classes)
240 labels = clf.predict(X_test) # (n_samples,)
241 ```
242
243 Datasets with more than the model's class capacity are handled automatically via **ECOC**;
244 tables wider than the feature limit are reduced by built-in
245 [**feature selection**](#feature-selection-wide-tables).
246 </details>
247
248 <details open>
249 <summary><b>Regression</b></summary>
250
251 ```python
252 from sklearn.datasets import load_diabetes
253 from sklearn.model_selection import train_test_split
254
255 from exaonetabular import EXAONETabularRegressor
256
257 X_train, X_test, y_train, y_test = train_test_split(
258 *load_diabetes(return_X_y=True), test_size=0.25, random_state=0
259 )
260
261 reg = EXAONETabularRegressor.from_pretrained(device="cuda:0") # download + verify + load
262
263 reg.fit(X_train, y_train) # y: (n,) real-valued, finite — fits ensemble weights
264 y_pred = reg.predict(X_test) # (n_samples,) float64 point estimates
265 ```
266
267 The head predicts a **999-quantile distribution** per row, which `predict` reduces to one number:
268 by default a **trimmed mean** — the trapezoidal average over the central 99.8% of the quantile
269 function, sorted first so crossed quantiles cannot flip the order. That targets the conditional
270 mean, which is what RMSE scores and what the median misses on skewed targets. To read the median
271 quantile instead, pass a `manifest=` whose `RegressionConfig` sets `point_estimate="median"`.
272
273 The ensemble members are then **weighted, not averaged**. `fit` holds out 20% of the support set,
274 predicts it from the rows that remain, and solves for non-negative member weights by least squares
275 (NNLS), rescaled to sum to one and blended 75/25 with the uniform `1/E`. Members whose preprocessing
276 rule suits your table earn more of the vote; non-negativity keeps the result a convex combination,
277 and the blend bounds how far a fit on a small split can stray from the uniform prior. This costs one
278 extra forward pass inside `fit` — `predict` stays single-pass.
279
280 The fit needs **2000 held-out rows** (`nnls_min_validation_rows`), so it engages from roughly 10k
281 support rows up; smaller tables log a warning and stay on the uniform mean, because a handful of
282 weights fitted against a few dozen rows is where the solve degenerates. Set
283 `RegressionConfig.member_weighting="uniform"` to switch it off entirely.
284
285 Targets are standardized against the fitted support set and the prediction is mapped back, so `y`
286 needs no scaling of your own — but it must be finite; `NaN`/`inf` targets raise.
287 Tables wider than **1024 columns** are narrowed by univariate `f_regression` (see
288 [Feature selection](#feature-selection-wide-tables)).
289 </details>
290
291 > **NaNs and categoricals.** `X` must be numeric — encode string/categorical columns to numeric
292 > codes before `fit` (e.g. a stable ordinal map), leaving unseen/missing values as `NaN`. The
293 > built-in preprocessor mean-imputes `NaN`s; it does not encode raw strings.
294
295 ### Overrides
296
297 `from_pretrained` accepts optional overrides without leaving the one-call path:
298
299 ```python
300 clf = EXAONETabularClassifier.from_pretrained(
301 device="cuda:0",
302 compute_dtype="bfloat16", # wider exponent range (default: "float16")
303 ensemble_count=8, seed=0, # runtime knobs
304 revision="main", # pin a specific Hub revision (tag or commit sha)
305 max_vram_bytes=24 << 30, # cap the GPU memory budget (see Out-of-memory below)
306 )
307
308 # Load your own weights of the same architecture — a local file or a Hub repo id.
309 # The released SHA-256 pin only applies to the released file, so it is not enforced
310 # here (a warning is logged); shapes, dtype, and finiteness are still validated.
311 clf = EXAONETabularClassifier.from_pretrained(weights="/path/to/my-classifier.safetensors")
312 ```
313
314 You can also redirect the weights without touching code via the environment:
315 `EXAONETABULAR_CLASSIFIER_WEIGHTS` / `EXAONETABULAR_REGRESSOR_WEIGHTS` (a local path or a repo id).
316
317 > **Precision.** The released weights are stored in **float32**. With the default
318 > `compute_dtype="float16"` they are cast to fp16 at load — the tested runtime path. fp16 is the
319 > default because it is the more precise of the two half formats at the same footprint and
320 > throughput — 10 mantissa bits to bf16's 7 — and this model's activations stay far from fp16's
321 > 65504 ceiling, so bf16's wider exponent range buys nothing here. The two score the same in our
322 > classification benchmarking; prefer `compute_dtype="bfloat16"` only if your inputs can
323 > drive activations to that ceiling. `compute_dtype="float32"` is a **CPU-only** path: part of
324 > the attention stack is pinned to the FlashAttention kernel, which implements fp16 and bf16
325 > only, so a float32 forward on a CUDA device fails with `RuntimeError: No available kernel`.
326
327 <details>
328 <summary><b>Advanced: fully custom checkpoint (explicit manifest)</b></summary>
329
330 `from_pretrained` is a thin layer over the low-level API. For a checkpoint with a **different
331 architecture**, describe it with an `InferenceManifest` and load it explicitly — this is the same
332 API the released presets are built from:
333
334 ```python
335 from huggingface_hub import hf_hub_download
336 from exaonetabular import (
337 EXAONETabularClassifier,
338 InferenceManifest,
339 ModelConfig,
340 RuntimeConfig,
341 load_classifier_checkpoint,
342 )
343
344 CKPT = hf_hub_download("your-org/your-repo", "your-classifier.safetensors")
345 manifest = InferenceManifest(
346 task="classification",
347 model=ModelConfig(class_capacity=10), # must match the checkpoint's class-head width
348 runtime=RuntimeConfig(ensemble_count=8, compute_dtype="float16", seed=0),
349 )
350
351 clf = EXAONETabularClassifier(manifest, device="cuda:0") # builds the model
352 load_classifier_checkpoint(CKPT, clf.model, manifest) # validates + loads weights
353 ```
354
355 A classification manifest carries a `ClassificationConfig` too. Leaving it off, as above, fills in
356 the defaults; `n_svd=0` therefore leaves support-SVD augmentation disabled. To opt in, import
357 `ClassificationConfig` and pass `classification=ClassificationConfig(n_svd=8)` to the manifest.
358 This changes inference preprocessing only and does not require different checkpoint weights.
359
360 Regression is analogous with `EXAONETabularRegressor`, `load_regressor_checkpoint`, and a
361 `RegressionConfig`. Two of its fields describe the checkpoint and must match it —
362 `quantile_count=999` and `decoder_hidden_width=384` — while `point_estimate`, the `n_svd`/`svd_*`
363 fields, and the `member_weighting`/`nnls_*` fields are readout and ensembling choices you can
364 change without touching the weights.
365
366 The frozen manifests the released estimators use live in `presets.py` and are reachable via
367 `released_manifest("classification" | "regression")`.
368 </details>
369
370
371 ### Feature selection (wide tables)
372
373 The classifier accepts tables of any width, but the model itself reads at most **100 columns**. When
374 `fit` receives a wider table, it chooses which columns to keep using the model's own attention —
375 there is no flag, and nothing to configure:
376
377 ```python
378 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0")
379 clf.fit(X_train, y_train) # X_train: (n, 5000) — selection runs here
380
381 clf.n_features_in_ # 5000 — the public width does not change
382 clf.selected_feature_indices_ # (100,) int64, the columns actually kept
383 clf.predict_proba(X_test) # still takes all 5000 columns
384 ```
385
386 **How it works.** One forward pass over a ≤512-row sample of the fitted table, with the
387 feature-attention blocks instrumented. Two signals are read per column — attention from the target
388 row, and the summed attention from the item-summary rows — each weighted by the value-vector norm so
389 the score reflects information actually routed through the attention path rather than raw attention
390 probability. The two are min-max normalized, averaged, and the top 100 columns are kept.
391
392 **What to expect.**
393
394 - Narrow tables (`n_features ≤ 100`) skip this entirely — the pass does not run.
395 - Selection is **internal**. `n_features_in_`, `predict`, and `predict_proba` all keep the original
396 width; the fitted column subset is reapplied for you.
397 - It costs one extra forward pass per `fit` on a wide table. A GPU is strongly recommended, and in
398 this version there is **no way to disable it**.
399 - Classification only — see below for the regressor.
400
401 The configuration is frozen in `config.py` as `FEATURE_SELECTION`. It belongs to the
402 architecture rather than to any one checkpoint — the scorers name the model's token layout, so
403 the same settings apply to every classifier checkpoint of this architecture.
404
405 **Optional support-SVD augmentation.** By default, the classifier does not append support-SVD
406 components (`ClassificationConfig.n_svd=0`). Set `n_svd` to a positive integer—for example,
407 `n_svd=8`—to append that many components to every ensemble member's features, with each member
408 projecting onto its own basis. Once enabled, this augmentation is unconditional: unlike the
409 regressor, classification has no small-table exemption or un-augmented comparison arm because
410 probability aggregation averages members rather than fitting member weights.
411
412 **The regressor narrows differently.** `EXAONETabularRegressor` reads up to **1024 columns** and
413 trims anything wider with univariate `f_regression` — an F-test against the target, so it costs no
414 extra forward pass and uses no attention. It then appends **16 support-SVD components** to every
415 ensemble member's features (`RegressionConfig.n_svd`), each member projecting onto its own basis,
416 so the model sees the kept columns plus that augmentation.
417
418 Two knobs control when that augmentation applies, both settled once in `fit` against the whole
419 support:
420
421 - **`svd_gate`** (default `False`) withholds it from small, narrow, all-numeric tables — fewer than
422 1000 rows *and* fewer than 10 columns *and* no categorical column — where a near-full-rank SVD
423 only restates the input. Off by default because the split below already prices the augmentation by
424 weight; set `True` to make the exemption an all-or-nothing decision instead.
425 - **`svd_split`** (default `True`) runs **two** ensembles instead of one, an un-augmented pass and
426 an augmented pass, and pools their members into a single prediction so the weight fit decides how
427 much the augmentation is worth rather than the gate deciding all-or-nothing. It doubles both the
428 member count and the **forwards, at predict as well as fit**. Both passes share the run's seed, so
429 a member differs across them only by the augmentation. With nothing to contrast against — exempted
430 by the gate, or `n_svd=0` — a split run collapses back to the single pass.
431
432 ### Controlling the GPU memory budget
433
434 Before running, the estimator measures the GPU, plans one execution strategy that
435 fits a memory **budget** (how many ensemble members run at once, how query rows
436 and feed-forward tokens are chunked, whether the support cache is offloaded), and
437 executes that plan. `max_vram_bytes` sets the budget explicitly:
438
439 ```python
440 clf = EXAONETabularClassifier.from_pretrained(device="cuda:0", max_vram_bytes=24 << 30)
441 ```
442
443 It is a **hard cap**, in bytes, and CUDA-only: the planner both *prefers* to stay
444 under it and treats it as the *feasibility* limit, so it will chunk more
445 aggressively to fit and will refuse — rather than quietly exceed it — a forward
446 whose smallest possible plan does not. Left unset, the budget is everything your
447 process can address: total VRAM minus what other processes already hold.
448
449 **To spend a proportion of the GPU, compute the bytes yourself** — there is no
450 separate fraction argument, because the proportion is only meaningful once you
451 choose what it is a proportion *of*:
452
453 ```python
454 import torch
455
456 free, total = torch.cuda.mem_get_info(0) # free = unused now, total = card capacity
457 clf = EXAONETabularClassifier.from_pretrained(
458 device="cuda:0",
459 max_vram_bytes=int(0.7 * free), # 70% of what is actually free right now
460 )
461 ```
462
463 > **Pick the denominator deliberately.** `total` is the card's capacity; `free` is
464 > what is unused at that moment. On a shared GPU a fraction of `total` can exceed
465 > what your process is able to obtain, which plans a forward that cannot run — use
466 > `free` unless you own the whole device. Note also that the planner already keeps
467 > a ~10% safety margin against the budget on the memory-heaviest build phases, so
468 > a budget of *B* is planned to roughly *0.9B*; there is no need to discount twice.
469
470 ### Out-of-memory and memory fragmentation
471
472 Large support sets on a memory-constrained GPU can trigger a CUDA out-of-memory
473 error. **The error is raised to you unchanged.** Inference plans once and runs
474 that plan; it does not catch the OOM, shrink the budget, and silently retry.
475 Recovering costs GPU time and is a policy decision — retry smaller, fall back to
476 CPU, fail the request — so it belongs to the caller:
477
478 ```python
479 try:
480 proba = clf.predict_proba(X)
481 except torch.cuda.OutOfMemoryError:
482 # Your policy: e.g. re-fit with a lower max_vram_bytes or ensemble_count.
483 ...
484 ```
485
486 Before concluding the model does not fit, check whether the failure is
487 **external fragmentation** rather than a true capacity limit. In the CUDA error,
488 compare the amount it *tried to allocate* against the `reserved but unallocated`
489 figure: when a large amount is reserved-but-unallocated yet a much smaller
490 allocation fails, the data would fit but the caching allocator cannot place a
491 single contiguous block — that is fragmentation, not lack of memory.
492
493 For that case, run with PyTorch's expandable-segments allocator. It lets the
494 allocator grow and coalesce segments, which largely removes contiguous-block
495 fragmentation:
496
497 ```bash
498 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python your_script.py
499 ```
500
501 > It is a **process-global** setting and must be present in the environment
502 > **before** CUDA initializes — set it when launching the process, not from inside
503 > Python after torch has already allocated. It changes only the allocator; results
504 > are unaffected.
505
506 If it still OOMs with expandable segments, the working set genuinely exceeds VRAM.
507 Reduce the footprint instead, roughly in order of cost to accuracy:
508
509 1. **Lower `max_vram_bytes`.** A smaller budget makes the planner chunk harder:
510 slower, but the same computation — chunking splits batch dimensions and does
511 not change the model. Chunked and unchunked results agree to numerical
512 tolerance rather than bit-for-bit, which is visible only in reduced precision.
513 2. **Lower `ensemble_count`** (a `from_pretrained` override) — fewer ensemble
514 members is directly less work and less memory, at some accuracy cost.
515 3. **Shrink the in-context support set** via the low-level
516 `RuntimeConfig(support_row_limit=…)` manifest path. This is the only lever on
517 the memory floor that grows with support rows, and the most costly to accuracy.
518 4. **Use a larger GPU.**
519
520
521 ## Available checkpoints
522
523 | File | Task | Head | Dtype | Notes |
524 |---|---|---|---|---|
525 | `exaone-tabular-classifier-v1_default.safetensors` | Classification | 10-class | float32 | `> class_capacity` classes handled automatically via ECOC |
526 | `exaone-tabular-regressor-v1_default.safetensors` | Regression | 999 quantiles | float32 | Read out as a trimmed mean over the quantiles; needs a `RegressionConfig` in its manifest |
527
528 Both live in the same Hub repository, and each estimator's `from_pretrained()` fetches its own file
529 — there is no shared dual-head checkpoint, and a classifier file will not load into the regressor.
530
531 Each checkpoint's architecture is **frozen** and must match its `InferenceManifest`; a mismatched
532 file (wrong keys, shapes, or dtype) fails loudly at load — never silently. The regression loader is
533 stricter still: the file must carry a `quantile_levels` buffer in float32 that equals
534 `linspace(1/1000, 999/1000, 999)` exactly, so a head of a different width or spacing is rejected
535 rather than silently reinterpreted.
536
537 `InferenceManifest.checkpoint_sha256` can additionally pin one exact file. The released manifests in
538 `presets.py` leave it `None`, so **every load — both tasks — logs a warning** saying the bytes were
539 not integrity-checked; the structural validation above still runs. Set it to pin one exact byte
540 stream, and a checkpoint whose digest differs is rejected.
541
542
543 ## Intended use
544
545 EXAONE Tabular is intended for **supervised tabular** classification and regression on structured
546 (row/column) data, for datasets within the tested sample/feature envelope. High-dimensional inputs
547 are handled by built-in [feature selection](#feature-selection-wide-tables); large support sets are
548 subsampled. Use of the released weights is limited to
549 **non-commercial research and educational** purposes by the EXAONE model license.
550
551 **Not intended for:** unstructured data (images, raw text, audio, video); inputs substantially
552 beyond the tested envelope, where accuracy and runtime are not guaranteed; any **commercial** use
553 of the released weights, or any use excluded by the [license](#license).
554
555
556 ## Limitation
557
558 **Class-Count Handling**.
559 The native classification head supports up to 10 classes. Datasets with larger label
560 spaces are handled through an ECOC-based decomposition at inference time. This procedure requires
561 multiple binary predictions and therefore increases inference cost as the number of classes grows. A class-
562 count-independent prediction head is a potential direction for future work.
563
564 **Large-Context Inference**.
565 Query chunking controls peak query-side memory because query predictions
566 are conditionally independent given the support set. However, the current inference wrapper recomputes
567 the support representations for each estimator and query chunk, introducing redundant computation when
568 either the ensemble size or the number of query chunks is large. The model already provides a support-side
569 caching path for row-axis attention, but this path is not yet used by the default chunked-inference wrapper.
570 Activating support-representation caching could reduce repeated computation across query chunks.
571 Support sets beyond the configured inference limit are currently subsampled. Potential future directions
572 include support-side representation and KV caching, context compression, representative-context selec-
573 tion, clustering-based support reduction, retrieval-based context construction, memory-efficient attention,
574 and adaptive support-set sampling. These methods require systematic evaluation of the trade-offs among
575 inference latency, memory consumption, support compression, and predictive performance.
576
577
578 ## License
579
580 Two licenses apply, to two different things:
581
582 - **The code in this repository** — the `exaonetabular` inference runtime — is released under the
583 [BSD-3-Clause-LG AI Research License](https://github.com/LGAI-Research/EXAONE-Tabular/blob/main/LICENSE), which permits commercial use.
584 - **The released model weights** are licensed separately under the **EXAONE AI Model License
585 Agreement 1.2 - NC**, which limits use to non-commercial research and education. The full terms
586 ship with the weights on the [Hugging Face repository](https://huggingface.co/LG-AI-Research/EXAONE-Tabular).
587
588 Installing this package therefore does not grant commercial rights to the weights it downloads.
589
590
591 ## Citation
592
593 If you find EXAONE Tabular useful in your research, please cite our technical report:
594
595 ```bibtex
596 @article{eo2026exaonetabular,
597 title = {EXAONE Tabular 1.0: Technical Report},
598 author = {Eo, Moonjung and Suh, Min-Kook and Cho, Hye-Seung and Kim, Jiwon and Kim, Seoyoon and Nam, Sangjun and Lee, Soonyoung},
599 year = {2026},
600 eprint = {2608.25774},
601 archivePrefix = {arXiv},
602 primaryClass = {cs.LG}
603 }
604 ```
605
606 ## Contact
607
608 LG AI Research Technical Support: contact_us@lgresearch.ai
609