Dataset Card for FinBench

Dataset Statistics

We present FinBench, a benchmark for evaluating the performance of machine learning models with both tabular data inputs and profile text inputs.

We first collect hundreds of financial datasets from the Kaggle platform and then screen out ten high-quality datasets for financial risk prediction. The screening criteria is based on the quantity and popularity, column meaningfulness, and the performance of baseline models on those datasets.

FinBench consists of three types of financial risks, i.e., default, fraud, and churn. We process the datasets in a unified data structure and provide an easy-loading API on Model Database.

Task Statistics

The following table reports the task description, dataset name (for datasets loading), the number and positive ratio of train/validation/test sets, the number of classification classes (all is 2), and the number of features.

Data Source

• Language: English

Dataset Structure

Data Fields

import datasets

datasets.Features(
    {
        "X_ml": [datasets.Value(dtype="float")],  # (The tabular data array of the current instance)
        "X_ml_unscale": [datasets.Value(dtype="float")],  # (Scaled tabular data array of the current instance)
        "y": datasets.Value(dtype="int64"),  # (The label / ground-truth)
        "num_classes": datasets.Value("int64"),  # (The total number of classes)
        "num_features": datasets.Value("int64"),  # (The total number of features)
        "num_idx": [datasets.Value("int64")],  # (The indices of the numerical datatype columns)
        "cat_idx": [datasets.Value("int64")],  # (The indices of the categorical datatype columns)
        "cat_dim": [datasets.Value("int64")],  # (The dimension of each categorical column)
        "cat_str": [[datasets.Value("string")]],  # (The category names of categorical columns)
        "col_name": [datasets.Value("string")],  # (The name of each column)
        "X_instruction_for_profile": datasets.Value("string"),  # instructions (from tabular data) for profiles
        "X_profile": datasets.Value("string"),  # customer profiles built from instructions via LLMs
    }
)

Data Loading

Model Database Login (Optional)

# OR run Model Database-cli login
from huggingface_hub import login

hf_token = "YOUR_ACCESS_TOKENS"  # https://Model Database.co/settings/tokens
login(token=hf_token)

Loading a Dataset

from datasets import load_dataset

# ds_name_list = ["cd1", "cd2", "ld1", "ld2", "ld3", "cf1", "cf2", "cc1", "cc2", "cc3"]
ds_name = "cd1"  # change the dataset name here
dataset = load_dataset("yuweiyin/FinBench", ds_name)

Loading the Splits

from datasets import load_dataset

ds_name = "cd1"  # change the dataset name here
dataset = load_dataset("yuweiyin/FinBench", ds_name)

train_set = dataset["train"] if "train" in dataset else []
validation_set = dataset["validation"] if "validation" in dataset else []
test_set = dataset["test"] if "test" in dataset else []

Loading the Instances

from datasets import load_dataset

ds_name = "cd1"  # change the dataset name here
dataset = load_dataset("yuweiyin/FinBench", ds_name)
train_set = dataset["train"] if "train" in dataset else []

for train_instance in train_set:
    X_ml = train_instance["X_ml"]  # List[float] (The tabular data array of the current instance)
    X_ml_unscale = train_instance["X_ml_unscale"]  # List[float] (Scaled tabular data array of the current instance)
    y = train_instance["y"]  # int (The label / ground-truth)
    num_classes = train_instance["num_classes"]  # int (The total number of classes)
    num_features = train_instance["num_features"]  # int (The total number of features)
    num_idx = train_instance["num_idx"]  # List[int] (The indices of the numerical datatype columns)
    cat_idx = train_instance["cat_idx"]  # List[int] (The indices of the categorical datatype columns)
    cat_dim = train_instance["cat_dim"]  # List[int] (The dimension of each categorical column)
    cat_str = train_instance["cat_str"]  # List[List[str]] (The category names of categorical columns)
    col_name = train_instance["col_name"]  # List[str] (The name of each column)
    X_instruction_for_profile = train_instance["X_instruction_for_profile"]  # instructions for building profiles
    X_profile = train_instance["X_profile"]  # customer profiles built from instructions via LLMs

Citation

• arXiv: https://arxiv.org/abs/2308.00065
• GitHub: https://github.com/YuweiYin/FinPT

@article{yin2023finbench,
  title   = {FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models},
  author  = {Yin, Yuwei and Yang, Yazheng and Yang, Jian and Liu, Qi},
  journal = {arXiv preprint arXiv:2308.00065},
  year    = {2023},
}