[2510.06377] Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data

Computer Science > Machine Learning arXiv:2510.06377 (cs) [Submitted on 7 Oct 2025 (v1), last revised 2 Mar 2026 (this version, v3)] Title:Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data Authors:Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos Kanatsoulis, Roshan Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, Jure Leskovec View a PDF of the paper titled Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data, by Rishabh Ranjan and 9 other authors View PDF Abstract:Pretrained transformers readily adapt to new sequence modeling tasks via zero-shot prompting, but relational domains still lack architectures that transfer across datasets and tasks. The core challenge is the diversity of relational data, with varying heterogeneous schemas, graph structures and functional dependencies. In this paper, we present the Relational Transformer (RT) architecture, which can be pretrained on diverse relational databases and directly applied to unseen datasets and tasks without task- or dataset-specific fine-tuning, or retrieval of in-context examples. RT (i) incorporates task specification via task table prompting, (ii) tokenizes cells with table/column metadata, (iii) is pretrained via masked token prediction, and (iv) utilizes a novel Relational Attention mechanism over columns, rows, and primary-foreign key links. Pretrained on RelBench datasets spanning tasks such as churn and sales forecasting, RT...