MicPie/unpredictable_wiki-openmoko-org

Dataset Card for "UnpredicTable-wiki-openmoko-org" - Dataset of Few-shot Tasks from Tables

Dataset Summary

The UnpredicTable dataset consists of web tables formatted as few-shot tasks for fine-tuning language models to improve their few-shot performance.

There are several dataset versions available:

• UnpredicTable-full: Starting from the initial WTC corpus of 50M tables, we apply our tables-to-tasks procedure to produce our resulting dataset, UnpredicTable-full, which comprises 413,299 tasks from 23,744 unique websites.

• UnpredicTable-unique: This is the same as UnpredicTable-full but filtered to have a maximum of one task per website. UnpredicTable-unique contains exactly 23,744 tasks from 23,744 websites.

• UnpredicTable-5k: This dataset contains 5k random tables from the full dataset.

• UnpredicTable data subsets based on a manual human quality rating (please see our publication for details of the ratings):

  • UnpredicTable-rated-low
  • UnpredicTable-rated-medium
  • UnpredicTable-rated-high

• UnpredicTable data subsets based on the website of origin:

  • UnpredicTable-baseball-fantasysports-yahoo-com
  • UnpredicTable-bulbapedia-bulbagarden-net
  • UnpredicTable-cappex-com
  • UnpredicTable-cram-com
  • UnpredicTable-dividend-com
  • UnpredicTable-dummies-com
  • UnpredicTable-en-wikipedia-org
  • UnpredicTable-ensembl-org
  • UnpredicTable-gamefaqs-com
  • UnpredicTable-mgoblog-com
  • UnpredicTable-mmo-champion-com
  • UnpredicTable-msdn-microsoft-com
  • UnpredicTable-phonearena-com
  • UnpredicTable-sittercity-com
  • UnpredicTable-sporcle-com
  • UnpredicTable-studystack-com
  • UnpredicTable-support-google-com
  • UnpredicTable-w3-org
  • UnpredicTable-wiki-openmoko-org
  • UnpredicTable-wkdu-org

• UnpredicTable data subsets based on clustering (for the clustering details please see our publication):

  • UnpredicTable-cluster00
  • UnpredicTable-cluster01
  • UnpredicTable-cluster02
  • UnpredicTable-cluster03
  • UnpredicTable-cluster04
  • UnpredicTable-cluster05
  • UnpredicTable-cluster06
  • UnpredicTable-cluster07
  • UnpredicTable-cluster08
  • UnpredicTable-cluster09
  • UnpredicTable-cluster10
  • UnpredicTable-cluster11
  • UnpredicTable-cluster12
  • UnpredicTable-cluster13
  • UnpredicTable-cluster14
  • UnpredicTable-cluster15
  • UnpredicTable-cluster16
  • UnpredicTable-cluster17
  • UnpredicTable-cluster18
  • UnpredicTable-cluster19
  • UnpredicTable-cluster20
  • UnpredicTable-cluster21
  • UnpredicTable-cluster22
  • UnpredicTable-cluster23
  • UnpredicTable-cluster24
  • UnpredicTable-cluster25
  • UnpredicTable-cluster26
  • UnpredicTable-cluster27
  • UnpredicTable-cluster28
  • UnpredicTable-cluster29
  • UnpredicTable-cluster-noise

Supported Tasks and Leaderboards

Since the tables come from the web, the distribution of tasks and topics is very broad. The shape of our dataset is very wide, i.e., we have 1000's of tasks, while each task has only a few examples, compared to most current NLP datasets which are very deep, i.e., 10s of tasks with many examples. This implies that our dataset covers a broad range of potential tasks, e.g., multiple-choice, question-answering, table-question-answering, text-classification, etc.

The intended use of this dataset is to improve few-shot performance by fine-tuning/pre-training on our dataset.

Languages

English

Dataset Structure

Data Instances

Each task is represented as a jsonline file and consists of several few-shot examples. Each example is a dictionary containing a field 'task', which identifies the task, followed by an 'input', 'options', and 'output' field. The 'input' field contains several column elements of the same row in the table, while the 'output' field is a target which represents an individual column of the same row. Each task contains several such examples which can be concatenated as a few-shot task. In the case of multiple choice classification, the 'options' field contains the possible classes that a model needs to choose from.

There are also additional meta-data fields such as 'pageTitle', 'title', 'outputColName', 'url', 'wdcFile'.

Data Fields

'task': task identifier

'input': column elements of a specific row in the table.

'options': for multiple choice classification, it provides the options to choose from.

'output': target column element of the same row as input.

'pageTitle': the title of the page containing the table.

'outputColName': output column name

'url': url to the website containing the table

'wdcFile': WDC Web Table Corpus file

Data Splits

The UnpredicTable datasets do not come with additional data splits.

Dataset Creation

Curation Rationale

Few-shot training on multi-task datasets has been demonstrated to improve language models' few-shot learning (FSL) performance on new tasks, but it is unclear which training tasks lead to effective downstream task adaptation. Few-shot learning datasets are typically produced with expensive human curation, limiting the scale and diversity of the training tasks available to study. As an alternative source of few-shot data, we automatically extract 413,299 tasks from diverse internet tables. We provide this as a research resource to investigate the relationship between training data and few-shot learning.

Source Data

Initial Data Collection and Normalization

We use internet tables from the English-language Relational Subset of the WDC Web Table Corpus 2015 (WTC). The WTC dataset tables were extracted from the July 2015 Common Crawl web corpus (http://webdatacommons.org/webtables/2015/EnglishStatistics.html). The dataset contains 50,820,165 tables from 323,160 web domains. We then convert the tables into few-shot learning tasks. Please see our publication for more details on the data collection and conversion pipeline.

Who are the source language producers?

The dataset is extracted from WDC Web Table Corpora.

Annotations

Annotation process

Manual annotation was only carried out for the UnpredicTable-rated-low, UnpredicTable-rated-medium, and UnpredicTable-rated-high data subsets to rate task quality. Detailed instructions of the annotation instructions can be found in our publication.

Who are the annotators?

Annotations were carried out by a lab assistant.

Personal and Sensitive Information

The data was extracted from WDC Web Table Corpora, which in turn extracted tables from the Common Crawl. We did not filter the data in any way. Thus any user identities or otherwise sensitive information (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history, etc.) might be contained in our dataset.

Considerations for Using the Data

Social Impact of Dataset

This dataset is intended for use as a research resource to investigate the relationship between training data and few-shot learning. As such, it contains high- and low-quality data, as well as diverse content that may be untruthful or inappropriate. Without careful investigation, it should not be used for training models that will be deployed for use in decision-critical or user-facing situations.

Discussion of Biases

Since our dataset contains tables that are scraped from the web, it will also contain many toxic, racist, sexist, and otherwise harmful biases and texts. We have not run any analysis on the biases prevalent in our datasets. Neither have we explicitly filtered the content. This implies that a model trained on our dataset may potentially reflect harmful biases and toxic text that exist in our dataset.

Other Known Limitations

No additional known limitations.

Additional Information

Dataset Curators

Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, Ethan Perez

Licensing Information

Apache 2.0

Citation Information

@misc{chan2022few,
  author = {Chan, Jun Shern and Pieler, Michael and Jao, Jonathan and Scheurer, Jérémy and Perez, Ethan},
  title = {Few-shot Adaptation Works with UnpredicTable Data},
  publisher={arXiv},
  year = {2022},
  url = {https://arxiv.org/abs/2208.01009}
}