Datasets:
Tasks:
Multiple Choice
Sub-tasks:
multiple-choice-coreference-resolution
Languages:
English
Multilinguality:
monolingual
Size Categories:
n<1K
Language Creators:
expert-generated
Annotations Creators:
expert-generated
Source Datasets:
extended|winograd_wsc
ArXiv:
License:
cc-by-4.0
sentence
string
| question
string
| options
sequence
| answer
string
|
|---|---|---|---|
"The city councilmen refused the demonstrators a permit because they feared violence." | "Who feared violence?" | [
"councilmen",
"demonstrators"
] | "councilmen" |
"The city councilmen refused the demonstrators a permit because they advocated violence." | "Who advocated violence?" | [
"councilmen",
"demonstrators"
] | "demonstrators" |
"The trophy doesn't fit into the brown suitcase because it's too small." | "What is too small?" | [
"suitcase",
"trophy"
] | "suitcase" |
"The trophy doesn't fit into the brown suitcase because it's too large." | "What is too large?" | [
"suitcase",
"trophy"
] | "trophy" |
"Joan made sure to thank Susan for all the help she had given." | "Who had given help?" | [
"Susan",
"Joan"
] | "Susan" |
"Joan made sure to thank Susan for all the help she had received." | "Who had received help?" | [
"Susan",
"Joan"
] | "Joan" |
"Paul tried to call George on the phone, but he wasn't successful." | "Who was not successful?" | [
"Paul",
"George"
] | "Paul" |
"Paul tried to call George on the phone, but he wasn't available." | "Who was not available?" | [
"Paul",
"George"
] | "George" |
"The lawyer asked the witness a question, but he was reluctant to answer it." | "Who was reluctant to answer the question?" | [
"witness",
"lawyer"
] | "witness" |
"The lawyer asked the witness a question, but he was reluctant to repeat it." | "Who was reluctant to repeat the question?" | [
"witness",
"lawyer"
] | "lawyer" |
"The delivery truck zoomed by the school bus because it was going so fast." | "What was going so fast?" | [
"truck",
"bus"
] | "truck" |
"The delivery truck zoomed by the school bus because it was going so slow." | "What was going so slow?" | [
"truck",
"bus"
] | "bus" |
"Frank felt vindicated when his longtime rival Bill revealed that he was the winner of the competition." | "Who was the winner of the competition?" | [
"Frank",
"Bill"
] | "Frank" |
"Frank felt crushed when his longtime rival Bill revealed that he was the winner of the competition." | "Who was the winner of the competition?" | [
"Frank",
"Bill"
] | "Bill" |
"The man couldn't lift his son because he was so weak." | "Who was weak?" | [
"man",
"son"
] | "man" |
"The man couldn't lift his son because he was so heavy." | "Who was heavy?" | [
"man",
"son"
] | "son" |
"The large ball crashed right through the table because it was made of steel." | "What was made of steel?" | [
"ball",
"table"
] | "ball" |
"The large ball crashed right through the table because it was made of styrofoam." | "What was made of styrofoam?" | [
"ball",
"table"
] | "table" |
"John couldn't see the stage with Billy in front of him because he is so short." | "Who is so short?" | [
"John",
"Billy"
] | "John" |
"John couldn't see the stage with Billy in front of him because he is so tall." | "Who is so tall?" | [
"John",
"Billy"
] | "Billy" |
"Tom threw his schoolbag down to Ray after he reached the top of the stairs." | "Who reached the top of the stairs?" | [
"Tom",
"Ray"
] | "Tom" |
"Tom threw his schoolbag down to Ray after he reached the bottom of the stairs." | "Who reached the bottom of the stairs?" | [
"Tom",
"Ray"
] | "Ray" |
"Although they ran at about the same speed, Sue beat Sally because she had such a good start." | "Who had a good start?" | [
"Sue",
"Sally"
] | "Sue" |
"Although they ran at about the same speed, Sue beat Sally because she had such a bad start." | "Who had a bad start?" | [
"Sue",
"Sally"
] | "Sally" |
"The sculpture rolled off the shelf because it wasn't anchored." | "What wasn't anchored?" | [
"sculpture",
"shelf"
] | "sculpture" |
"The sculpture rolled off the shelf because it wasn't level." | "What wasn't level?" | [
"sculpture",
"shelf"
] | "shelf" |
"Sam's drawing was hung just above Tina's and it did look much better with another one below it." | "Which drawing looked better?" | [
"Sam's",
"Tina's"
] | "Sam's" |
"Sam's drawing was hung just above Tina's and it did look much better with another one above it." | "Which drawing looked better?" | [
"Sam's",
"Tina's"
] | "Tina's" |
"Anna did a lot better than her good friend Lucy on the test because she had studied so hard." | "Who studied hard?" | [
"Anna",
"Lucy"
] | "Anna" |
"Anna did a lot worse than her good friend Lucy on the test because she had studied so hard." | "Who studied hard?" | [
"Anna",
"Lucy"
] | "Lucy" |
"The firemen arrived after the police because they were coming from so far away." | "Who came from far away?" | [
"firemen",
"police"
] | "firemen" |
"The firemen arrived before the police because they were coming from so far away." | "Who came from far away?" | [
"firemen",
"police"
] | "police" |
"Frank was upset with Tom because the toaster he had bought from him didn't work." | "Who had bought the toaster?" | [
"Frank",
"Tom"
] | "Frank" |
"Frank was upset with Tom because the toaster he had sold to him didn't work." | "Who had sold the toaster?" | [
"Frank",
"Tom"
] | "Tom" |
"Jim yelled at Kevin because he was so upset." | "Who was upset?" | [
"Jim",
"Kevin"
] | "Jim" |
"Jim comforted Kevin because he was so upset." | "Who was upset?" | [
"Jim",
"Kevin"
] | "Kevin" |
"The sack of potatoes had been placed above the bag of flour, so it had to be moved first." | "What had to be moved first?" | [
"sack",
"bag"
] | "sack" |
"The sack of potatoes had been placed below the bag of flour, so it had to be moved first." | "What had to be moved first?" | [
"sack",
"bag"
] | "bag" |
"Pete envies Martin because he is very successful." | "Who is very successful?" | [
"Martin",
"Pete"
] | "Martin" |
"Pete envies Martin although he is very successful." | "Who is very successful?" | [
"Martin",
"Pete"
] | "Pete" |
"I was trying to balance the bottle upside down on the table, but I couldn't do it because it was so top-heavy." | "What was top-heavy?" | [
"bottle",
"table"
] | "bottle" |
"I was trying to balance the bottle upside down on the table, but I couldn't do it because it was so uneven." | "What was uneven?" | [
"bottle",
"table"
] | "table" |
"I spread the cloth on the table in order to protect it." | "To protect what?" | [
"table",
"cloth"
] | "table" |
"I spread the cloth on the table in order to display it." | "To display what?" | [
"table",
"cloth"
] | "cloth" |
"The seniors were bullying the freshmen, so we rescued them." | "Which students did we rescue?" | [
"freshmen",
"seniors"
] | "freshmen" |
"The seniors were bullying the freshmen, so we punished them." | "Which students did we punish?" | [
"freshmen",
"seniors"
] | "seniors" |
"I poured water from the bottle into the cup until it was full." | "What was full?" | [
"cup",
"bottle"
] | "cup" |
"I poured water from the bottle into the cup until it was empty." | "What was empty?" | [
"cup",
"bottle"
] | "bottle" |
"Susan knows all about Ann's personal problems because she is nosy." | "Who is nosy?" | [
"Susan",
"Anne"
] | "Susan" |
"Susan knows all about Ann's personal problems because she is indiscreet." | "Who is indiscreet?" | [
"Susan",
"Anne"
] | "Anne" |
"Susan knew that Ann's son had been in a car accident, so she told her about it." | "Who told whom?" | [
"Susan",
"Ann"
] | "Susan" |
"Susan knew that Ann's son had been in a car accident, because she told her about it." | "Who told whom?" | [
"Susan",
"Ann"
] | "Ann" |
"The police left the house and went into the garage where they found the murder weapon." | "Where did they find the murder weapon?" | [
"garage",
"house"
] | "garage" |
"The police left the house and went into the garage after they found the murder weapon." | "Where did they find the murder weapon?" | [
"garage",
"house"
] | "house" |
"The painting in Mark's living room shows an oak tree. It is to the right of the bookcase." | "What is to the right of the bookcase?" | [
"painting",
"tree"
] | "painting" |
"The painting in Mark's living room shows an oak tree. It is to the right of a house." | "What is to the right of a house?" | [
"painting",
"tree"
] | "tree" |
"That is the gap in the wall. You can see the garden through it." | "You can see the garden through what?" | [
"gap",
"wall"
] | "gap" |
"That is the gap in the wall. You can see the garden behind it." | "You can see the garden behind what?" | [
"gap",
"wall"
] | "wall" |
"The drain is clogged with hair. It has to be cleaned." | "What has to be cleaned?" | [
"drain",
"hair"
] | "drain" |
"The drain is clogged with hair. It has to be removed." | "What has to be removed?" | [
"drain",
"hair"
] | "hair" |
"The meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was short, so it worked out." | "What was short?" | [
"meeting",
"train"
] | "meeting" |
"The meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was delayed, so it worked out." | "What was delayed?" | [
"meeting",
"train"
] | "train" |
"There is a pillar between me and the stage, and I can't see it." | "What can't I see?" | [
"stage",
"pillar"
] | "stage" |
"There is a pillar between me and the stage, and I can't see around it." | "What can't I see around?" | [
"stage",
"pillar"
] | "pillar" |
"They broadcast an announcement, but a subway came into the station and I couldn't hear it." | "What couldn't I hear?" | [
"announcement",
"subway"
] | "announcement" |
"They broadcast an announcement, but a subway came into the station and I couldn't hear over it." | "What couldn't I hear over?" | [
"announcement",
"subway"
] | "subway" |
"In the middle of the outdoor concert, the rain started falling, and it continued until 10." | "What continued until 10?" | [
"rain",
"outdoor concert"
] | "rain" |
"In the middle of the outdoor concert, the rain started falling, but it continued until 10." | "What continued until 10?" | [
"rain",
"outdoor concert"
] | "outdoor concert" |
"I used an old rag to clean the knife, and then I put it in the drawer." | "What did I put in the drawer?" | [
"knife",
"rag"
] | "knife" |
"I used an old rag to clean the knife, and then I put it in the trash." | "What did I put in the trash?" | [
"knife",
"rag"
] | "rag" |
"Ann asked Mary what time the library closes, but she had forgotten." | "Who had forgotten?" | [
"Mary",
"Ann"
] | "Mary" |
"Ann asked Mary what time the library closes, because she had forgotten." | "Who had forgotten?" | [
"Mary",
"Ann"
] | "Ann" |
"I took the water bottle out of the backpack so that it would be lighter." | "What would be lighter?" | [
"backpack",
"bottle"
] | "backpack" |
"I took the water bottle out of the backpack so that it would be handy." | "What would be handy?" | [
"backpack",
"bottle"
] | "bottle" |
"I couldn't put the pot on the shelf because it was too high." | "What was too high?" | [
"shelf",
"pot"
] | "shelf" |
"I couldn't put the pot on the shelf because it was too tall." | "What was too tall?" | [
"shelf",
"pot"
] | "pot" |
"I'm sure that the map will show the building; it is very famous." | "What is famous?" | [
"building",
"map"
] | "building" |
"I'm sure that the map will show the building; it is very good." | "What is good?" | [
"building",
"map"
] | "map" |
"Bob paid for Charlie's college education. He is very generous." | "Who is generous?" | [
"Bob",
"Charlie"
] | "Bob" |
"Bob paid for Charlie's college education. He is very grateful." | "Who is grateful?" | [
"Bob",
"Charlie"
] | "Charlie" |
Dataset Card for The modified Winograd Schema Challenge (MWSC)
Dataset Summary
Examples taken from the Winograd Schema Challenge modified to ensure that answers are a single word from the context. This Modified Winograd Schema Challenge (MWSC) ensures that scores are neither inflated nor deflated by oddities in phrasing.
Supported Tasks and Leaderboards
Languages
Dataset Structure
Data Instances
default
- Size of downloaded dataset files: 0.02 MB
- Size of the generated dataset: 0.04 MB
- Total amount of disk used: 0.06 MB
An example looks as follows:
{
"sentence": "The city councilmen refused the demonstrators a permit because they feared violence.",
"question": "Who feared violence?",
"options": [ "councilmen", "demonstrators" ],
"answer": "councilmen"
}
Data Fields
The data fields are the same among all splits.
default
sentence: astringfeature.question: astringfeature.options: alistofstringfeatures.answer: astringfeature.
Data Splits
| name | train | validation | test |
|---|---|---|---|
| default | 80 | 82 | 100 |
Dataset Creation
Curation Rationale
Source Data
Initial Data Collection and Normalization
Who are the source language producers?
Annotations
Annotation process
Who are the annotators?
Personal and Sensitive Information
Considerations for Using the Data
Social Impact of Dataset
Discussion of Biases
Other Known Limitations
Additional Information
Dataset Curators
Licensing Information
Our code for running decaNLP has been open sourced under BSD-3-Clause.
We chose to restrict decaNLP to datasets that were free and publicly accessible for research, but you should check their individual terms if you deviate from this use case.
From the Winograd Schema Challenge:
Both versions of the collections are licenced under a Creative Commons Attribution 4.0 International License.
Citation Information
If you use this in your work, please cite:
@article{McCann2018decaNLP,
title={The Natural Language Decathlon: Multitask Learning as Question Answering},
author={Bryan McCann and Nitish Shirish Keskar and Caiming Xiong and Richard Socher},
journal={arXiv preprint arXiv:1806.08730},
year={2018}
}
Contributions
Thanks to @thomwolf, @lewtun, @ghomasHudson, @lhoestq for adding this dataset.
- Downloads last month
- 685