Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger

Accepted at ICML 2026

---

This is the official code for the paper "Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models". A research codebase for understanding inference dynamics in tabular in-context learning models (TabPFN, TabICL, LimiX, …) across OpenML benchmarks.

---

Repository structure

.
├── Experiments/                    # Main experiment code
│   ├── main.py                     # the main code for saving the results.
│   ├── fine_tuning_exp.py          # the code for tabular logit lens (pretrained indivudal decoders)
│   ├── util.py                     # some utilities (metrics, preprocessing, probing, …)
│   ├── configs/                    # Per-model experiment configs.
│   │   ├── tabpfn_v1/
│   │   ├── tabpfn_v2/
│   │   ├── tabpfn_v2_5/
│   │   ├── tabicl/
│   │   ├── limix_2m/
│   │   ├── limix_16m/
│   │   ├── nanotabpfn/
│   │   └── ...
│   ├── plots/                      # plots the results
│   └── results/                    # Output results (gitignored)
│
├── FoundationModels/               # Edited model packages
│   ├── TabPFN_v1/
│   ├── TabPFN_v2/
│   ├── TabPFN_v2_5/
│   ├── TabICL/
│   ├── Limix/
│   ├── NanoTabPFN/
│   └── weights/                    # Pre-trained model weights and indiviudal decoders weights 
│
├── Notebooks/                      # prototyping notebooks
│
├── requirements.txt
└── README.md

dependecy

1. Install UV

install UV:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.bashrc

# Create a venv (stored locally in the project folder)
uv venv --python 3.11
source .venv/bin/activate

# Install and run
uv pip install -r requirements.txt

2. Install model packages

Each model lives under FoundationModels/ (and better to be installed in editable mode):

uv pip install -e FoundationModels/TabPFN_v1
uv pip install -e FoundationModels/TabPFN_v2
uv pip install -e FoundationModels/TabPFN_v2_5
uv pip install -e FoundationModels/TabICL
uv pip install -e FoundationModels/Limix

Place the model checkpoint and fine-tuned decoders weights in FoundationModels/weights, to use the models please check the README in FoundationModels/

Experiment Configs

Each config file defines a different experimental condition applied to the model's layers:

Config	Description
c0	Default model — standard forward pass, no layer interventions
c1	Representation analysis — probes intermediate layer representations
c2	Layer skipping — evaluates the effect of skipping individual layers
c3	Layer repetition — evaluates the effect of repeating individual layers
c4	Layer swapping — evaluates the effect of swapping pairs of layers

Example: Running the experiment

change directory to Experiments/ .

cd Experiments

Single task

python main.py \
  --task_id 363619 \
  --config tabpfn_v2.config_c0 \
  --output_root_dir ./results

Licensing

This repository contains modified versions of several third-party packages under FoundationModels/. Each retains its original license. All modifications are made for academic research purposes only.

Package	License	Original Authors
TabPFN_v1	Apache 2.0	University of Freiburg (Hollmann, Müller, Eggensperger, Hutter)
TabPFN_v2	Prior Labs License v1.0	Prior Labs GmbH
TabPFN_v2_5	Prior Labs License v1.2	Prior Labs GmbH
TabICL	BSD 3-Clause	Soda team @ Inria
Limix	Apache-2.0	LimiX Authors (StableAI)

Note: TabPFN_v2 requires attribution — any distribution or publication that uses this code or model weights must prominently display "Built with TabPFN" per Section 10 of the Prior Labs License v1.0.

Note: TabPFN_v2_5 requires attribution — any distribution or publication that uses this code or model weights must prominently display "Built with PriorLabs-TabPFN" per Section 10 of the Prior Labs License v1.2.

The experiment code in Experiments/, and Notebooks/ is released under the MIT License.

Acknowledgements

Built with TabPFN · Built with PriorLabs-TabPFN