BlockTabBench: Evaluating Tabular Models Across Block's Machine Learning Ecosystem

Introduction

Tabular machine learning is entering a transition similar to what natural language processing experienced during the emergence of large language models, with foundation models becoming an increasingly important part of the research landscape [1, 2, 3, 4]. As a result, standardized evaluations such as TabArena [5] and BeyondArena [6] have become valuable tools for understanding the relative efficacy of different modeling approaches and for driving reproducible progress across the field.

At Block, we have a large and diverse collection of datasets used to train models that identify fraudulent or high-risk activity across our products, accumulated across many years of operating production machine learning systems. These datasets are based on financial activity with very different characteristics than existing public datasets such as substantially higher row and column counts, varying amounts of imbalance, and temporal drift. While recent efforts such as BeyondArena represent a shift toward these more realistic properties, public evaluations typically aim to measure tabular machine learning performance in a domain general sense and thus do not fully capture the scale or characteristics of Block’s specific financial domain.

We developed BlockTabBench to assess tabular machine learning performance across Block’s machine learning problems. We annotate individual datasets with metadata such as the business event of interest (e.g., every Square payment, every Cash App deposit), the problem domain (e.g., scams, account takeovers), and attributes from the dataset (e.g., row and column count, imbalance, temporal drift). We benchmarked various models on each dataset to understand the strengths and weaknesses of each approach. Overall, we found:

• Tabular foundation models that rely on in-context learning are highly performant when training data is limited but lose their advantage as the dataset size grows
• Standard deep learning approaches for tabular data scale with millions of rows, outperforming both tree-based methods and tabular foundation models on larger datasets
• The advantage of tree models is concentrated on problems where every approach performs well, and narrows as the achievable ceiling lowers

Block’s Machine Learning Ecosystem

At Block, we are often interested in scanning an event stream like Cash App transactions for a problem like a scam or account takeover. We call the former the trigger and the latter the concern. A single trigger typically fans out to many concerns, with the same event stream scored by several models trained with different labels.

We sampled over one hundred datasets across all of our triggers and concerns and built BlockTabBench2026, an evaluation that is representative of Block’s ecosystem of production machine learning models as of 2026. BlockTabBench2026 contains datasets spanning approximately a few thousand rows to several million rows and column counts from a few hundred into the mid-thousands.

Concerns vary in how often they occur, which gives us tasks ranging from rare-event detection to nearly balanced classification as well as in how quickly they can be confirmed, which determines how much labeled data is available.

Every dataset is split out-of-time, matching how the corresponding production models were developed (on historical data) and deployed (on unseen future data). Public evaluations have traditionally assumed random splits, whereas our results reflect a more realistic setting where there may be significant drift between the train and test distributions.

Evaluating Tabular Models

We benchmarked various modeling approaches on BlockTabBench2026 to understand how they performed overall as well as how dataset properties contributed to each model’s performance. We were interested in comparing tree models (XGBoost [7]) against modern deep learning approaches for tabular data (TabM [8]) and tabular foundation models that rely on in-context learning for classification (TabFM [4]).

We used the area under the Precision-Recall curve (PR-AUC) to determine a winner for each dataset. We chose PR-AUC because production models are deployed at a decision threshold that varies by concern, trading off precision and recall according to the cost of acting on a false positive versus missing a true positive. PR-AUC evaluates a model across all such thresholds rather than committing to a single operating point.

For each dataset, we evaluated multiple random seeds with XGBoost and TabM undergoing hyperparameter tuning and TabFM using 50K sampled rows but otherwise fixed hyperparameters (32-member ensemble, top 500 features via XGBoost importance). Benchmarking was performed on de-identified data with engineered features in Block's internal infrastructure.

Counting the datasets each model wins, we find XGBoost to perform the best overall (41% win rate), closely followed by TabM (37% win rate), and finally TabFM (22% win rate). Breaking down the wins by dataset properties like train rows and problem difficulty, measured as the best PR-AUC any model reaches, paints a more nuanced picture where different models excel in different regimes. For each model, we fit a logistic regression predicting its wins from these properties.

TabFM’s in-context learning excels under data scarcity, similar to early findings in the large language modeling literature around the effectiveness of few-shot prompting [9]. This makes TabFM useful when labeled data is limited such as with newly emerging fraud patterns or other situations that require classification from few targeted examples. As the number of training rows increases, TabM’s standard gradient-based training overtakes other modeling approaches, consistent with deep learning scaling laws [10, 11]. In contrast to TabFM’s utility under label constraints, TabM is useful for problems with high label volume such as those where labels are continuously generated by a process like a manual review workforce investigating flagged transactions.

We also find that XGBoost's wins are concentrated on the problems where every approach performs, but its advantage narrows as the ceiling falls. This finding suggests that XGBoost is a reasonable default if it performs near ceiling, but for problems far from ceiling, exploring alternatives could pay off.

Summary

Constructing a benchmark that captures the nature of Block’s machine learning ecosystem allows us to draw conclusions about the efficacy of different tabular models similar to how evaluations are used to benchmark the advances in large language models. We find that tabular foundation models – which have recently exceeded XGBoost and more standard tabular deep learning approaches on public benchmarks like TabArena – only maintain this lead in the limited data regime. While standard tabular deep learning approaches excelled when large amounts of training data were available, it is possible that tabular foundation models can maintain their advantage via fine-tuning [12] or training from scratch on large amounts of synthetic data that better represents the activity seen across Block’s services.

References

[1] Hollmann, Noah, et al. "Accurate predictions on small data with a tabular foundation model." Nature 637.8045 (2025): 319-326.

[2] Grinsztajn, Léo, et al. "TabPFN-3: Technical report." arXiv preprint arXiv

.13986 (2026).

[3] Qu, Jingang, et al. "TabICLv2: A better, faster, scalable, and open tabular foundation model." Proceedings of the 43rd International Conference on Machine Learning (2026).

[4] Kong, Weihao, and Abhimanyu Das. "Introducing TabFM: a zero-shot foundation model for tabular data." https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data (2026).

[5] Erickson, Nick, et al. "TabArena: A living benchmark for machine learning on tabular data." Advances in Neural Information Processing Systems 38 (2026).

[6] Purucker, Lennart, et al. "Beyond IID: How general are tabular foundation models, really?" arXiv preprint arXiv

.30410 (2026).

[7] Chen, Tianqi, and Carlos Guestrin. "XGBoost: A scalable tree boosting system." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).

[8] Gorishniy, Yury, et al. "TabM: Advancing tabular deep learning with parameter-efficient ensembling." Proceedings of the 13th International Conference on Learning Representations (2025).

[9] Brown, Tom B., et al. "Language models are few-shot learners." Advances in Neural Information Processing Systems 33 (2020).

[10] Hestness, Joel, et al. "Deep learning scaling is predictable, empirically." arXiv preprint arXiv

.00409 (2017).

[11] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv

.08361 (2020).

[12] Rubachev, Ivan, et al. "On finetuning tabular foundation models." arXiv preprint arXiv

.08982 (2025).