Benchmarks

TiRex-2's reported results are reproducible against two public leaderboards. Full instructions live in the repository next to the benchmark code:

• GIFT-Eval — reproduce results on the GIFT-Eval leaderboard.
• fev-bench — reproduce results on the fev-bench leaderboard.

GIFT-Eval

Download the dataset once:

pixi run -e example-cu128 huggingface-cli download Salesforce/GiftEval --repo-type=dataset --local-dir PATH_TO_SAVE

Run the benchmark, choosing a model type:

pixi run -e example-cu128 python examples/gifteval/run_gifteval.py </path/to/gifteval_storage> pretrained

• pretrained loads NX-AI/TiRex-2-gifteval-pretrain.
• zero-shot loads NX-AI/TiRex-2-gifteval-zs.

By default (--eval-mode multivariate) the native multivariate target is kept intact and scored jointly, which exercises TiRex-2's cross-variate path but is not directly comparable to the public GIFT-Eval leaderboard. Pass --eval-mode univariate to split every multivariate dataset into independent univariate channels, matching the official leaderboard protocol:

pixi run -e example-cu126 python examples/gifteval/run_gifteval.py \
    </path/to/gifteval_storage> <ckpt_dir> --eval-mode univariate

An interactive notebook is also available: start pixi run notebook and open examples/gifteval/gifteval.ipynb.

fev-bench

Optionally pre-download the data:

pixi run -e example-cu128 huggingface-cli download autogluon/fev_datasets --repo-type=dataset --local-dir </path/to/fevbench/store>

Run the benchmark — this always loads NX-AI/TiRex-2-fevbench from Hugging Face:

pixi run fevbench [/path/to/fevbench_storage] [--tasks examples/fevbench/tasks.yaml]

If the storage path is omitted, the dataset is downloaded at runtime and cached under $HOME/.cache.