"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models

Ricardo Knauer, Mario Koddenbrock, Raphael Wallsberger, Nicholas M. Brisson, Georg N. Duda +3 more
9/27/2024
cs.AIcs.CLcs.LG

Official code: ml-lab-htw/llm-trees (16 stars, MIT, last commit 2025-08-05).

Abstract

Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can even surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform better than data-driven tree-based embeddings on average. Our decision tree induction and embedding approaches can therefore serve as new knowledge-driven baselines for data-driven machine learning methods in the low-data regime. Furthermore, they offer ways to harness the rich world knowledge within LLMs for tabular machine learning tasks. Our code and results are available at https://github.com/ml-lab-htw/llm-trees.

View on arXivView PDF

Code Implementations(1)

ml-lab-htw/llm-treesOfficial100%
MIT

Official repo: “Oh LLM, I’m Asking Thee, Please Give Me a Decision Tree”: Zero-Shot Decision Tree Induction and Embedding with Large Language Models

164TeX, JuliaDec 12, 20241 years agoMIT

Similar Papers

Zero-Shot Decision Tree Construction via Large Language Models

94% match2025

FORESTLLM: Large Language Models Make Random Forest Great on Few-shot Tabular Learning

87% match2026

Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data

86% match2025

Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning

85% match2024

Programmatic Representation Learning with Language Models

85% match1 repo62025

Through the Thicket: A Study of Number-Oriented LLMs derived from Random Forest Models

83% match2024

Cite this paper

@article{knauer2024llm,
  title  = {"Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models},
  author = {Ricardo Knauer and Mario Koddenbrock and Raphael Wallsberger and Nicholas M. Brisson and Georg N. Duda and Deborah Falla and David W. Evans and Erik Rodner},
  year   = {2024},
  eprint = {2409.18594},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.18594v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.18594.svg)](https://codeofpaper.com/paper/2409.18594)

Discussion