sileod/deberta-v3-base-tasksource-nli
Zero-Shot Classification
•
Updated
•
12.1k
•
73
Datasets:
Languages:
English
Multilinguality:
monolingual
Size Categories:
10K<n<100K
Language Creators:
other
Annotations Creators:
expert-generated
Source Datasets:
extended|other
License:
unknown
sentence1
string
| sentence2
string
| idx
int32
0
1.1k
| label
class label
2 classes
|
|---|---|---|---|
"The cat sat on the mat." | "The cat did not sit on the mat." | 0 | 1
(not_entailment) |
"The cat did not sit on the mat." | "The cat sat on the mat." | 1 | 1
(not_entailment) |
"When you've got no snow, it's really hard to learn a snow sport so we looked at all the different ways I could mimic being on snow without actually being on snow." | "When you've got snow, it's really hard to learn a snow sport so we looked at all the different ways I could mimic being on snow without actually being on snow." | 2 | 1
(not_entailment) |
"When you've got snow, it's really hard to learn a snow sport so we looked at all the different ways I could mimic being on snow without actually being on snow." | "When you've got no snow, it's really hard to learn a snow sport so we looked at all the different ways I could mimic being on snow without actually being on snow." | 3 | 1
(not_entailment) |
"Out of the box, Ouya supports media apps such as Twitch.tv and XBMC media player." | "Out of the box, Ouya doesn't support media apps such as Twitch.tv and XBMC media player." | 4 | 1
(not_entailment) |
"Out of the box, Ouya doesn't support media apps such as Twitch.tv and XBMC media player." | "Out of the box, Ouya supports media apps such as Twitch.tv and XBMC media player." | 5 | 1
(not_entailment) |
"Out of the box, Ouya supports media apps such as Twitch.tv and XBMC media player." | "Out of the box, Ouya supports Twitch.tv and XBMC media player." | 6 | 0
(entailment) |
"Out of the box, Ouya supports Twitch.tv and XBMC media player." | "Out of the box, Ouya supports media apps such as Twitch.tv and XBMC media player." | 7 | 0
(entailment) |
"Considering this definition, it is surprising to find frequent use of sarcastic language in opinionated user generated content." | "Considering this definition, it is not surprising to find frequent use of sarcastic language in opinionated user generated content." | 8 | 1
(not_entailment) |
"Considering this definition, it is not surprising to find frequent use of sarcastic language in opinionated user generated content." | "Considering this definition, it is surprising to find frequent use of sarcastic language in opinionated user generated content." | 9 | 1
(not_entailment) |
"The new gaming console is affordable." | "The new gaming console is unaffordable." | 10 | 1
(not_entailment) |
"The new gaming console is unaffordable." | "The new gaming console is affordable." | 11 | 1
(not_entailment) |
"Brexit is an irreversible decision, Sir Mike Rake, the chairman of WorldPay and ex-chairman of BT group, said as calls for a second EU referendum were sparked last week." | "Brexit is a reversible decision, Sir Mike Rake, the chairman of WorldPay and ex-chairman of BT group, said as calls for a second EU referendum were sparked last week." | 12 | 1
(not_entailment) |
"Brexit is a reversible decision, Sir Mike Rake, the chairman of WorldPay and ex-chairman of BT group, said as calls for a second EU referendum were sparked last week." | "Brexit is an irreversible decision, Sir Mike Rake, the chairman of WorldPay and ex-chairman of BT group, said as calls for a second EU referendum were sparked last week." | 13 | 1
(not_entailment) |
"We built our society on unclean energy." | "We built our society on clean energy." | 14 | 1
(not_entailment) |
"We built our society on clean energy." | "We built our society on unclean energy." | 15 | 1
(not_entailment) |
"Pursuing a strategy of nonviolent protest, Gandhi took the administration by surprise and won concessions from the authorities." | "Pursuing a strategy of violent protest, Gandhi took the administration by surprise and won concessions from the authorities." | 16 | 1
(not_entailment) |
"Pursuing a strategy of violent protest, Gandhi took the administration by surprise and won concessions from the authorities." | "Pursuing a strategy of nonviolent protest, Gandhi took the administration by surprise and won concessions from the authorities." | 17 | 1
(not_entailment) |
"Pursuing a strategy of nonviolent protest, Gandhi took the administration by surprise and won concessions from the authorities." | "Pursuing a strategy of protest, Gandhi took the administration by surprise and won concessions from the authorities." | 18 | 0
(entailment) |
"Pursuing a strategy of protest, Gandhi took the administration by surprise and won concessions from the authorities." | "Pursuing a strategy of nonviolent protest, Gandhi took the administration by surprise and won concessions from the authorities." | 19 | 1
(not_entailment) |
"And if both apply, they are essentially impossible." | "And if both apply, they are essentially possible." | 20 | 1
(not_entailment) |
"And if both apply, they are essentially possible." | "And if both apply, they are essentially impossible." | 21 | 1
(not_entailment) |
"Writing Java is not too different from programming with handcuffs." | "Writing Java is similar to programming with handcuffs." | 22 | 0
(entailment) |
"Writing Java is similar to programming with handcuffs." | "Writing Java is not too different from programming with handcuffs." | 23 | 0
(entailment) |
"The market is about to get harder, but not impossible to navigate." | "The market is about to get harder, but possible to navigate." | 24 | 0
(entailment) |
"The market is about to get harder, but possible to navigate." | "The market is about to get harder, but not impossible to navigate." | 25 | 0
(entailment) |
"Even after now finding out that it's animal feed, I won't ever stop being addicted to Flamin' Hot Cheetos." | "Even after now finding out that it's animal feed, I will never stop being addicted to Flamin' Hot Cheetos." | 26 | 0
(entailment) |
"Even after now finding out that it's animal feed, I will never stop being addicted to Flamin' Hot Cheetos." | "Even after now finding out that it's animal feed, I won't ever stop being addicted to Flamin' Hot Cheetos." | 27 | 0
(entailment) |
"He did not disagree with the party's position, but felt that if he resigned, his popularity with Indians would cease to stifle the party's membership." | "He agreed with the party's position, but felt that if he resigned, his popularity with Indians would cease to stifle the party's membership." | 28 | 0
(entailment) |
"He agreed with the party's position, but felt that if he resigned, his popularity with Indians would cease to stifle the party's membership." | "He did not disagree with the party's position, but felt that if he resigned, his popularity with Indians would cease to stifle the party's membership." | 29 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would not be unexpected." | 30 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would not be unexpected." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | 31 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, it would be expected to negatively impact the pipeline results." | 32 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, it would be expected to negatively impact the pipeline results." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | 33 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, it would not be unexpected for it to negatively impact the pipeline results." | 34 | 0
(entailment) |
"If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, it would not be unexpected for it to negatively impact the pipeline results." | "If the pipeline tokenization scheme does not correspond to the one that was used when a model was created, a negative impact on the pipeline results would be expected." | 35 | 0
(entailment) |
"The water is too hot." | "The water is too cold." | 36 | 1
(not_entailment) |
"The water is too cold." | "The water is too hot." | 37 | 1
(not_entailment) |
"Falcon Heavy is the largest rocket since NASA's Saturn V booster, which was used for the Moon missions in the 1970s." | "Falcon Heavy is the smallest rocket since NASA's Saturn V booster, which was used for the Moon missions in the 1970s." | 38 | 1
(not_entailment) |
"Falcon Heavy is the smallest rocket since NASA's Saturn V booster, which was used for the Moon missions in the 1970s." | "Falcon Heavy is the largest rocket since NASA's Saturn V booster, which was used for the Moon missions in the 1970s." | 39 | 1
(not_entailment) |
"Adenoiditis symptoms often persist for ten or more days, and often include pus-like discharge from nose." | "Adenoiditis symptoms often pass within ten days or less, and often include pus-like discharge from nose." | 40 | 1
(not_entailment) |
"Adenoiditis symptoms often pass within ten days or less, and often include pus-like discharge from nose." | "Adenoiditis symptoms often persist for ten or more days, and often include pus-like discharge from nose." | 41 | 1
(not_entailment) |
"In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite difficult to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 42 | 1
(not_entailment) |
"In example (1) it is quite difficult to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 43 | 1
(not_entailment) |
"In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite easy to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 44 | 0
(entailment) |
"In example (1) it is quite easy to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 45 | 0
(entailment) |
"In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite important to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 46 | 1
(not_entailment) |
"In example (1) it is quite important to see the exaggerated positive sentiment used in order to convey strong negative feelings." | "In example (1) it is quite straightforward to see the exaggerated positive sentiment used in order to convey strong negative feelings." | 47 | 1
(not_entailment) |
"Some dogs like to scratch their ears." | "Some animals like to scratch their ears." | 48 | 0
(entailment) |
"Some animals like to scratch their ears." | "Some dogs like to scratch their ears." | 49 | 1
(not_entailment) |
"Cruz has frequently derided as "amnesty" various plans that confer legal status or citizenship on people living in the country illegally." | "Cruz has frequently derided as "amnesty" various bills that confer legal status or citizenship on people living in the country illegally." | 50 | 1
(not_entailment) |
"Cruz has frequently derided as "amnesty" various bills that confer legal status or citizenship on people living in the country illegally." | "Cruz has frequently derided as "amnesty" various plans that confer legal status or citizenship on people living in the country illegally." | 51 | 0
(entailment) |
"Most of the graduates of my program have moved on to other things because the jobs suck." | "Some of the graduates of my program have moved on to other things because the jobs suck." | 52 | 0
(entailment) |
"Some of the graduates of my program have moved on to other things because the jobs suck." | "Most of the graduates of my program have moved on to other things because the jobs suck." | 53 | 1
(not_entailment) |
"In many developed areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | "In many areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | 54 | 0
(entailment) |
"In many areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | "In many developed areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | 55 | 1
(not_entailment) |
"We consider some context words as positive examples and sample negatives at random from the dictionary." | "We consider some words as positive examples and sample negatives at random from the dictionary." | 56 | 0
(entailment) |
"We consider some words as positive examples and sample negatives at random from the dictionary." | "We consider some context words as positive examples and sample negatives at random from the dictionary." | 57 | 1
(not_entailment) |
"We consider some context words as positive examples and sample negatives at random from the dictionary." | "We consider all context words as positive examples and sample many negatives at random from the dictionary." | 58 | 1
(not_entailment) |
"We consider all context words as positive examples and sample many negatives at random from the dictionary." | "We consider some context words as positive examples and sample negatives at random from the dictionary." | 59 | 1
(not_entailment) |
"We consider some context words as positive examples and sample negatives at random from the dictionary." | "We consider many context words as positive examples and sample negatives at random from the dictionary." | 60 | 1
(not_entailment) |
"We consider many context words as positive examples and sample negatives at random from the dictionary." | "We consider some context words as positive examples and sample negatives at random from the dictionary." | 61 | 0
(entailment) |
"We consider all context words as positive examples and sample negatives at random from the dictionary." | "We consider all words as positive examples and sample negatives at random from the dictionary." | 62 | 1
(not_entailment) |
"We consider all words as positive examples and sample negatives at random from the dictionary." | "We consider all context words as positive examples and sample negatives at random from the dictionary." | 63 | 0
(entailment) |
"All dogs like to scratch their ears." | "All animals like to scratch their ears." | 64 | 1
(not_entailment) |
"All animals like to scratch their ears." | "All dogs like to scratch their ears." | 65 | 0
(entailment) |
"Cruz has frequently derided as "amnesty" any plan that confers legal status or citizenship on people living in the country illegally." | "Cruz has frequently derided as "amnesty" any bill that confers legal status or citizenship on people living in the country illegally." | 66 | 0
(entailment) |
"Cruz has frequently derided as "amnesty" any bill that confers legal status or citizenship on people living in the country illegally." | "Cruz has frequently derided as "amnesty" any plan that confers legal status or citizenship on people living in the country illegally." | 67 | 1
(not_entailment) |
"Most of the graduates of my program have moved on to other things because the jobs suck." | "None of the graduates of my program have moved on to other things because the jobs suck." | 68 | 1
(not_entailment) |
"None of the graduates of my program have moved on to other things because the jobs suck." | "Most of the graduates of my program have moved on to other things because the jobs suck." | 69 | 1
(not_entailment) |
"Most of the graduates of my program have moved on to other things because the jobs suck." | "All of the graduates of my program have moved on to other things because the jobs suck." | 70 | 1
(not_entailment) |
"All of the graduates of my program have moved on to other things because the jobs suck." | "Most of the graduates of my program have moved on to other things because the jobs suck." | 71 | 1
(not_entailment) |
"In all areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | "In all developed areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | 72 | 0
(entailment) |
"In all developed areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | "In all areas, human activity has changed the form of river channels, altering magnitudes and frequencies of flooding." | 73 | 1
(not_entailment) |
"Tom and Adam were whispering in the theater." | "Tom and Adam were whispering quietly in the theater." | 74 | 0
(entailment) |
"Tom and Adam were whispering quietly in the theater." | "Tom and Adam were whispering in the theater." | 75 | 0
(entailment) |
"Tom and Adam were whispering in the theater." | "Tom and Adam were whispering loudly in the theater." | 76 | 1
(not_entailment) |
"Tom and Adam were whispering loudly in the theater." | "Tom and Adam were whispering in the theater." | 77 | 0
(entailment) |
"Prior to the dance, which is voluntary, students are told to fill out a card by selecting five people they want to dance with." | "Prior to the dance, which is voluntary, students are told to fill out a card by selecting five different people they want to dance with." | 78 | 0
(entailment) |
"Prior to the dance, which is voluntary, students are told to fill out a card by selecting five different people they want to dance with." | "Prior to the dance, which is voluntary, students are told to fill out a card by selecting five people they want to dance with." | 79 | 0
(entailment) |
"Notifications about Farmville and other crap had become unbearable, then the shift to the non-chronological timeline happened and the content from your friends started to be replaced by ads and other cringy wannabe-viral campaigns." | "Notifications about Farmville and other crappy apps had become unbearable, then the shift to the non-chronological timeline happened and the content from your friends started to be replaced by ads and other cringy wannabe-viral campaigns." | 80 | 0
(entailment) |
"Notifications about Farmville and other crappy apps had become unbearable, then the shift to the non-chronological timeline happened and the content from your friends started to be replaced by ads and other cringy wannabe-viral campaigns." | "Notifications about Farmville and other crap had become unbearable, then the shift to the non-chronological timeline happened and the content from your friends started to be replaced by ads and other cringy wannabe-viral campaigns." | 81 | 0
(entailment) |
"Chicago City Hall is the official seat of government of the City of Chicago." | "Chicago City Hall is the official seat of government of Chicago." | 82 | 0
(entailment) |
"Chicago City Hall is the official seat of government of Chicago." | "Chicago City Hall is the official seat of government of the City of Chicago." | 83 | 0
(entailment) |
"The question generation aspect is unique to our formulation, and corresponds roughly to identifying what semantic role labels are present in previous formulations of the task." | "The question generation aspect is unique to our formulation, and corresponds roughly to identifying what semantic role labels are present in previous other formulations of the task." | 84 | 0
(entailment) |
"The question generation aspect is unique to our formulation, and corresponds roughly to identifying what semantic role labels are present in previous other formulations of the task." | "The question generation aspect is unique to our formulation, and corresponds roughly to identifying what semantic role labels are present in previous formulations of the task." | 85 | 0
(entailment) |
"John ate pasta for dinner." | "John ate pasta for supper." | 86 | 0
(entailment) |
"John ate pasta for supper." | "John ate pasta for dinner." | 87 | 0
(entailment) |
"John ate pasta for dinner." | "John ate pasta for breakfast." | 88 | 1
(not_entailment) |
"John ate pasta for breakfast." | "John ate pasta for dinner." | 89 | 1
(not_entailment) |
"House Speaker Paul Ryan was facing problems from fellow Republicans dissatisfied with his leadership." | "House Speaker Paul Ryan was facing problems from fellow Republicans unhappy with his leadership." | 90 | 0
(entailment) |
"House Speaker Paul Ryan was facing problems from fellow Republicans unhappy with his leadership." | "House Speaker Paul Ryan was facing problems from fellow Republicans dissatisfied with his leadership." | 91 | 0
(entailment) |
"House Speaker Paul Ryan was facing problems uniquely from fellow Republicans dissatisfied with his leadership." | "House Speaker Paul Ryan was facing problems uniquely from fellow Republicans supportive of his leadership." | 92 | 1
(not_entailment) |
"House Speaker Paul Ryan was facing problems uniquely from fellow Republicans supportive of his leadership." | "House Speaker Paul Ryan was facing problems uniquely from fellow Republicans dissatisfied with his leadership." | 93 | 1
(not_entailment) |
"I can actually see him climbing into a Lincoln saying this." | "I can actually see him getting into a Lincoln saying this." | 94 | 0
(entailment) |
"I can actually see him getting into a Lincoln saying this." | "I can actually see him climbing into a Lincoln saying this." | 95 | 0
(entailment) |
"I can actually see him climbing into a Lincoln saying this." | "I can actually see him climbing into a Mazda saying this." | 96 | 1
(not_entailment) |
"I can actually see him climbing into a Mazda saying this." | "I can actually see him climbing into a Lincoln saying this." | 97 | 1
(not_entailment) |
"The villain is the character who tends to have a negative effect on other characters." | "The villain is the character who tends to have a negative impact on other characters." | 98 | 0
(entailment) |
"The villain is the character who tends to have a negative impact on other characters." | "The villain is the character who tends to have a negative effect on other characters." | 99 | 0
(entailment) |
Dataset Card for "super_glue"
Dataset Summary
SuperGLUE (https://super.gluebenchmark.com/) is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard.
BoolQ (Boolean Questions, Clark et al., 2019a) is a QA task where each example consists of a short passage and a yes/no question about the passage. The questions are provided anonymously and unsolicited by users of the Google search engine, and afterwards paired with a paragraph from a Wikipedia article containing the answer. Following the original work, we evaluate with accuracy.
Supported Tasks and Leaderboards
Languages
Dataset Structure
Data Instances
axb
- Size of downloaded dataset files: 0.03 MB
- Size of the generated dataset: 0.24 MB
- Total amount of disk used: 0.27 MB
An example of 'test' looks as follows.
axg
- Size of downloaded dataset files: 0.01 MB
- Size of the generated dataset: 0.05 MB
- Total amount of disk used: 0.06 MB
An example of 'test' looks as follows.
boolq
- Size of downloaded dataset files: 4.12 MB
- Size of the generated dataset: 10.40 MB
- Total amount of disk used: 14.52 MB
An example of 'train' looks as follows.
cb
- Size of downloaded dataset files: 0.07 MB
- Size of the generated dataset: 0.20 MB
- Total amount of disk used: 0.28 MB
An example of 'train' looks as follows.
copa
- Size of downloaded dataset files: 0.04 MB
- Size of the generated dataset: 0.13 MB
- Total amount of disk used: 0.17 MB
An example of 'train' looks as follows.
Data Fields
The data fields are the same among all splits.
axb
sentence1: astringfeature.sentence2: astringfeature.idx: aint32feature.label: a classification label, with possible values includingentailment(0),not_entailment(1).
axg
premise: astringfeature.hypothesis: astringfeature.idx: aint32feature.label: a classification label, with possible values includingentailment(0),not_entailment(1).
boolq
question: astringfeature.passage: astringfeature.idx: aint32feature.label: a classification label, with possible values includingFalse(0),True(1).
cb
premise: astringfeature.hypothesis: astringfeature.idx: aint32feature.label: a classification label, with possible values includingentailment(0),contradiction(1),neutral(2).
copa
premise: astringfeature.choice1: astringfeature.choice2: astringfeature.question: astringfeature.idx: aint32feature.label: a classification label, with possible values includingchoice1(0),choice2(1).
Data Splits
axb
| test | |
|---|---|
| axb | 1104 |
axg
| test | |
|---|---|
| axg | 356 |
boolq
| train | validation | test | |
|---|---|---|---|
| boolq | 9427 | 3270 | 3245 |
cb
| train | validation | test | |
|---|---|---|---|
| cb | 250 | 56 | 250 |
copa
| train | validation | test | |
|---|---|---|---|
| copa | 400 | 100 | 500 |
Dataset Creation
Curation Rationale
Source Data
Initial Data Collection and Normalization
Who are the source language producers?
Annotations
Annotation process
Who are the annotators?
Personal and Sensitive Information
Considerations for Using the Data
Social Impact of Dataset
Discussion of Biases
Other Known Limitations
Additional Information
Dataset Curators
Licensing Information
Citation Information
@inproceedings{clark2019boolq,
title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
author={Clark, Christopher and Lee, Kenton and Chang, Ming-Wei, and Kwiatkowski, Tom and Collins, Michael, and Toutanova, Kristina},
booktitle={NAACL},
year={2019}
}
@article{wang2019superglue,
title={SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems},
author={Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R},
journal={arXiv preprint arXiv:1905.00537},
year={2019}
}
Note that each SuperGLUE dataset has its own citation. Please see the source to
get the correct citation for each contained dataset.
Contributions
Thanks to @thomwolf, @lewtun, @patrickvonplaten for adding this dataset.
- Downloads last month
- 264,813
Homepage:
github.com
Size of downloaded dataset files:
58.4 MB
Size of the auto-converted Parquet files:
103 MB
Number of rows:
196,313
