README.md
5.7 KB · 170 lines · markdown Raw
1 ---
2 license: other
3 license_name: tabfm-non-commercial-v1.0
4 license_link: https://huggingface.co/google/tabfm-1.0.0-pytorch/blob/main/LICENSE
5 library_name: tabfm
6 pipeline_tag: tabular-classification
7 tags:
8 - tabular
9 - tabular-regression
10 - zero-shot
11 - in-context-learning
12 - pytorch
13 - foundation-model
14 ---
15
16 # TabFM 1.0.0 (PyTorch)
17
18 TabFM is a zero-shot tabular foundation model from Google Research. It supports
19 classification and regression on structured/tabular data with mixed numerical and
20 categorical columns, requiring no fine-tuning or hyperparameter search - training
21 examples are passed as context and predictions are made in a single forward pass.
22
23 This repository contains the **PyTorch** weights. For the JAX/Flax weights see
24 [google/tabfm-1.0.0-jax](https://huggingface.co/google/tabfm-1.0.0-jax).
25
26 ## Getting Started
27
28 ```bash
29 pip install tabfm[pytorch]
30 ```
31
32 **Classification:**
33
34 ```python
35 from tabfm import TabFMClassifier, tabfm_v1_0_0_pytorch as tabfm_v1_0_0
36
37 model = tabfm_v1_0_0.load(model_type="classification")
38 clf = TabFMClassifier(model=model)
39 clf.fit(X_train, y_train)
40 probs = clf.predict_proba(X_test)
41 ```
42
43 **Regression:**
44
45 ```python
46 from tabfm import TabFMRegressor, tabfm_v1_0_0_pytorch as tabfm_v1_0_0
47
48 model = tabfm_v1_0_0.load(model_type="regression")
49 reg = TabFMRegressor(model=model)
50 reg.fit(X_train, y_train)
51 preds = reg.predict(X_test)
52 ```
53
54 You can also load directly using the HuggingFace Hub API:
55
56 ```python
57 from tabfm.src.pytorch.tabfm_v1_0_0 import TabFM_HF
58
59 clf_model = TabFM_HF.from_pretrained("google/tabfm-1.0.0-pytorch", subfolder="classification")
60 reg_model = TabFM_HF.from_pretrained("google/tabfm-1.0.0-pytorch", subfolder="regression")
61 ```
62
63 ### Available Checkpoints
64
65 | Subfolder | Task | `is_classifier` |
66 |-----------|------|-----------------|
67 | `classification/` | Classification (up to 10 classes) | `True` |
68 | `regression/` | Regression | `False` |
69
70 ## Developers and Affiliations
71
72 Developed by the [Google Research](https://research.google) team.
73
74 ## Intended Use
75
76 - Tabular data with numerical and/or categorical columns
77 - Binary and multiclass classification (up to 10 classes)
78 - Regression on continuous targets
79 - Zero-shot inference: no dataset-specific training or hyperparameter tuning
80 - Works with DataFrames (pandas) or numpy arrays
81
82 ## Not Intended For
83
84 - Images, audio, video, or raw text
85 - More than 10 output classes (hard model limit)
86 - Tasks requiring task-specific fine-tuning
87 - Non-tabular structured data (graphs, sequences)
88 - Commercial use (see License below)
89
90 ## Model Architecture
91
92 TabFM uses alternating row and column attention to capture both feature interactions
93 and row-level patterns:
94
95 1. **Column attention** (Set Transformer): embeds each cell using Fourier features and
96 a per-group linear projection, then aggregates across rows via induced self-attention
97 2. **Row compression**: CLS tokens summarise each row into a dense vector via row-level
98 attention with Rotary Position Embedding (RoPE)
99 3. **ICL Transformer**: a 24-block causal transformer operates over the compressed row
100 vectors, treating training rows as context and outputting predictions for test rows
101
102 Key hyperparameters:
103
104 | Parameter | Value |
105 |-----------|-------|
106 | Embedding dim | 256 |
107 | Column attention blocks | 3 (4 heads, 256 induced points) |
108 | Row attention blocks | 3 (8 heads, 8 CLS tokens) |
109 | ICL transformer blocks | 24 (8 heads) |
110 | Feed-forward factor | 4 |
111 | Max classes | 10 |
112 | Activation | SwiGLU |
113 | Fourier features | 32 frequencies |
114
115 ## Training Data and Priors
116
117 TabFM was trained on hundreds of millions of **synthetic** datasets generated
118 dynamically using structural causal models (SCMs). Synthetic data was chosen due to
119 the scarcity of diverse, high-quality open-source tabular datasets and to avoid
120 privacy/licensing concerns with real-world industrial data. The SCM prior encodes
121 inductive biases about causal structure and feature relationships typical in tabular
122 tasks.
123
124 ## Performance
125
126 TabFM was evaluated on [TabArena](https://tabarena.ai) across 51 datasets
127 (38 classification, 13 regression). In zero-shot mode - a single forward pass with no
128 hyperparameter search - TabFM outperforms heavily-tuned supervised baselines including
129 gradient-boosted trees. The `TabFMClassifier.ensemble()` preset (feature crosses,
130 SVD features, NNLS blending) yields further improvements.
131
132 See the [Google Research blog post](https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/) for full benchmark details.
133
134 ## Ethical Considerations
135
136 TabFM was trained entirely on synthetic data. Performance on specific real-world
137 domains, minority groups, or edge distributions is not fully characterised. Users
138 should evaluate the model on held-out data representative of their use case before
139 deploying in high-stakes settings.
140
141 ## Limitations
142
143 - **Max 10 classes** for classification (hard architectural limit)
144 - Memory usage scales with the number of training rows (all rows are passed as context)
145 - Optimised for tables up to 500 features; behaviour on very wide tables may degrade
146 - Performance is not guaranteed to match task-specific, fine-tuned models on all datasets
147 - Not an officially supported Google product
148
149 ## License
150
151 The model weights in this repository are released under the
152 **TabFM Non-Commercial License v1.0** - see [LICENSE](https://huggingface.co/google/tabfm-1.0.0-pytorch/blob/main/LICENSE). The source code is
153 Apache 2.0 licensed via [google-research/tabfm](https://github.com/google-research/tabfm).
154
155 ## Version
156
157 1.0.0
158
159 ## Citation
160
161 ```bibtex
162 @article{tabfm2026,
163 title = {TabFM: A Zero-Shot Foundation Model for Tabular Data},
164 author = {Google Research},
165 year = {2026},
166 url = {https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/}
167 }
168 ```
169
170