README.md
7.6 KB · 194 lines · markdown Raw
1 ---
2 license: apache-2.0
3 library_name: synthefy-nori
4 pipeline_tag: tabular-regression
5 tags:
6 - tabular
7 - tabular-regression
8 - tabular-foundation-model
9 - in-context-learning
10 - synthetic-data
11 - pytorch
12 ---
13
14 <p align="center">
15 <img src="synthefy_nori_banner.png" alt="Nori" width="100%">
16 </p>
17
18 # Nori
19
20 **Nori** is a tabular foundation model for **regression** via in-context
21 learning (ICL). Given a few labeled rows as context, it predicts on new query rows in a
22 **single forward pass**, with no task-specific training or fine-tuning. The model is
23 trained **entirely on synthetic data**.
24
25 - **Documentation:** https://docs.synthefy.com/nori/
26 - **Repository:** https://github.com/Synthefy/synthefy-nori
27 - **Library:** `pip install synthefy-nori`
28 - **Checkpoint:** `nori.pt` (this repo)
29 - **Parameters:** ~5.9M
30 - **License:** Apache-2.0
31
32 ## Results
33
34 Mean and median R² of the base model across 96 regression tasks from three
35 public benchmark suites (single H200, up to 50K context rows per dataset):
36
37 | Suite | Datasets | Mean R² | Median R² |
38 |-------|---------:|--------:|----------:|
39 | TabArena | 13 | 0.8117 | 0.8757 |
40 | TALENT | 72 | 0.7569 | 0.8802 |
41 | OpenML | 11 | 0.6373 | 0.5856 |
42 | **Overall** | **96** | **0.7506** | **0.8702** |
43
44 Large-N / long-context tables (common in TabArena) are the current focus of the
45 large-table training stages. These numbers are reproducible end-to-end with one
46 command — see [Reproducing these numbers](https://github.com/Synthefy/synthefy-nori#reproducing-these-numbers).
47
48 > **Thinking** is an inference-time reasoning extension that improves these
49 > numbers further. Details are forthcoming.
50
51 ## Use it from your AI coding assistant
52
53 Paste this into Claude Code, Cursor, or any AI coding assistant and it will wire
54 Nori into your own project:
55
56 ````text
57 Look at my code/task/report here and figure out where Nori would best fit — it's
58 Synthefy's tabular foundation model, a drop-in scikit-learn estimator that predicts
59 a continuous target by in-context learning: no training loop, no hyperparameters,
60 and it uses the GPU automatically when one's available (CPU otherwise).
61
62 1. Install it with this project's package manager
63 (e.g. `uv add synthefy-nori`, or `pip install -U synthefy-nori`).
64
65 2. Use it wherever a tabular regression / prediction step fits:
66
67 ```python
68 from synthefy_nori import NoriRegressor
69
70 reg = NoriRegressor(model="nori-6m") # downloads these weights from the Hub on first predict
71 reg.fit(X_train, y_train) # stores your rows as context — no training happens
72 y_pred = reg.predict(X_test) # point predictions (predictive-distribution mean)
73
74 # Prediction intervals come free — no conformal/quantile add-ons:
75 lo, mid, hi = reg.predict(X_test, output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
76 ```
77
78 X is a numeric feature matrix (encode categoricals as ordinals/one-hot, leave
79 missing values as NaN, no scaling needed); y is a finite continuous target. If I
80 already have a model, wire Nori up alongside it on the same train/test split and
81 metric so I can compare them. If the best place to plug Nori in isn't obvious,
82 show me where you'd put it and confirm with me before making changes.
83
84 Going deeper: synthefy-nori ships a ready-made nori-regression skill for AI coding
85 assistants with vetted recipes — calibrated prediction intervals, honest baseline
86 comparison under fixed CV, SHAP/PDP interpretability, and leak-safe one-step
87 time-series forecasting. Read and follow it if relevant:
88 https://github.com/Synthefy/synthefy-nori/tree/main/.claude/skills/nori-regression
89 ````
90
91 ## Usage
92
93 ```bash
94 pip install synthefy-nori
95 ```
96
97 ```python
98 from sklearn.datasets import load_diabetes
99 from sklearn.model_selection import train_test_split
100 from synthefy_nori import NoriRegressor
101
102 X, y = load_diabetes(return_X_y=True)
103 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
104
105 model = NoriRegressor() # downloads these weights from the Hub on first use
106 model.fit(X_train, y_train) # "fit" just stores the labeled rows as context
107 pred = model.predict(X_test) # predictions in a single forward pass, no training
108 ```
109
110 It uses a GPU when one is available and falls back to CPU. A one-shot helper skips the
111 object entirely:
112
113 ```python
114 from synthefy_nori import predict
115 pred = predict(X_train, y_train, X_test, task="regression")
116 ```
117
118 `predict` follows the `TabPFNRegressor.predict` contract: pass `output_type="mean"`
119 (default), `"median"`, or `"mode"` to choose the point estimate drawn from the model's
120 predictive distribution.
121
122 To run from a local checkpoint instead of the Hub default, pass a path:
123
124 ```python
125 model = NoriRegressor(model_path="path/to/checkpoint.pt")
126 ```
127
128 This checkpoint is **public**: the first inference call downloads and caches it
129 automatically, with no token and no access request. A Hugging Face token (read scope)
130 is only worth setting if you hit anonymous download rate limits — provide it via
131 `export HF_TOKEN=hf_...`, `hf auth login`, or `NoriRegressor(token="hf_...")`.
132
133 ## How it works
134
135 ### Architecture
136
137 A **FeaturesTransformer (~5.9M parameters)** that alternates two kinds of attention:
138
139 - **Feature attention** learns relationships between columns.
140 - **Sample attention** learns relationships between rows (context and query).
141 - **In-context learning**: predictions condition on labeled context rows, with no
142 gradient updates at inference.
143
144 Key config: 16 transformer layers, embed_dim 128, hidden 384, 2 heads, the **v2-lite**
145 block (SwiGLU + RMSNorm + pre-norm), features grouped in pairs (`features_per_group=2`),
146 with **column-specific y-aware** feature attention. Features are encoded with RBF
147 embeddings; missing values are handled natively via learned mask embeddings. The
148 regression head predicts a full distribution over 999 quantiles (pinball loss).
149
150 ### Synthetic data
151
152 The model never sees real data during training. Its capability comes from a diverse
153 synthetic data generator covering real-world tabular regimes:
154
155 - **Structural Causal Models (SCM)**: hierarchical DAGs with 8 edge-function types
156 (MLP, decision tree, piecewise-linear, polynomial, periodic, RBF, log/exp, conv1d).
157 - **Regression priors**: 9 target families (dense/sparse linear, GAM, interactions,
158 random MLP, random tree, radial/RBF, Fourier features, chained trigonometric).
159 - **Realism augmentations**: discretized features, noise features, correlated blocks,
160 structural missingness, label noise.
161 - **Learnability filter**: an ExtraTrees signal-quality filter rejects unlearnable
162 datasets so training compute is spent on learnable tasks.
163
164 Training runs entirely on synthetic data and **trains to completion** — there is no
165 real-data validation in the loop, so no benchmark data is needed to train and no eval
166 signal influences checkpoint selection. See the
167 [training guide](https://github.com/Synthefy/synthefy-nori/blob/main/docs/training.md)
168 for the full curriculum recipe.
169
170 ## Intended use & limitations
171
172 - **Intended for** small-to-medium tabular regression where in-context learning is
173 attractive (no per-task training).
174 - **Limitations:** the current gap vs the best baselines is on **large-N / long-context**
175 TabArena datasets; dense O(N²) sample attention bounds practical context size. Very large
176 tables are the focus of the large-table training stages.
177
178 ## Citation
179
180 ```bibtex
181 @software{synthefy_nori_2026,
182 title = {Nori: A Tabular Foundation Model Trained on Synthetic Data},
183 author = {Synthefy},
184 year = {2026},
185 url = {https://github.com/Synthefy/synthefy-nori}
186 }
187 ```
188
189 ## License
190
191 Apache-2.0. See
192 [LICENSE](https://github.com/Synthefy/synthefy-nori/blob/main/LICENSE) and
193 [NOTICE](https://github.com/Synthefy/synthefy-nori/blob/main/NOTICE).
194