uuid
string
sub
string
obj
string
pred
string
obj_label
string
masked_sentence
string
negated
string
"d4f11631dde8a43beda613ec845ff7d1"
"alive"
"think"
"HasSubevent"
"think"
"One of the things you do when you are alive is [MASK]."
""
"4b0c94af3a9339e4e5a3f9cbc057fac8"
"analyse"
"paralysis"
"HasSubevent"
"paralysis"
"Something that might happen when you analyse something is [MASK]."
""
"63471b3167a6a8769d9f3748ca5d0e0e"
"analysing"
"analysis"
"HasSubevent"
"analysis"
"Something that might happen while analysing something is [MASK]."
""
"67d690e232c0dcff0f1dab22037471bd"
"analysing"
"education"
"HasSubevent"
"education"
"Something that might happen while analysing something is [MASK]."
""
"f36e54ad99a0353e5a51e0e3487c489d"
"analysing"
"idea"
"HasSubevent"
"idea"
"Something that might happen while analysing something is coming up with a new [MASK]."
""
"371bb74ee9929945d917abcc35baecd9"
"analysing"
"smell"
"HasSubevent"
"smell"
"Something you might do while analysing something is [MASK] it."
""
"cb76eaf72a9c1850f250ab8e05e24b07"
"angry"
"yelling"
"HasSubevent"
"yelling"
"One of the things you do when you are angry is [MASK]."
""
"671bfa2bc5354764fc906d4f91773030"
"asleep"
"dreaming"
"HasSubevent"
"Dreaming"
"The story "[MASK]" has the step "I fell asleep."."
""
"2ec3c3dd9269da6c8b0ae0bee4858681"
"awake"
"sneezing"
"HasSubevent"
"sneezing"
"Something that might happen when you awake is [MASK]."
""
"4be96e8873d9715017e3a39cf8bda18b"
"awake"
"stretch"
"HasSubevent"
"stretch"
"The statement "Something that might happen when you awake is you [MASK]." is true because After sleeping, the body has lain relatively motionless."
""
"ba91527f9a49e97b9921e5085f62b6f8"
"awake"
"think"
"HasSubevent"
"think"
"If you want to [MASK] then you should be awake."
""
"f48a8099beae39ef800cbd4d68cefd34"
"awaking"
"alarm"
"HasSubevent"
"alarm"
"Something that might happen while awaking is realising that you haven't heard the [MASK] and that you are late for work."
""
"2bfdc20590295ab8ef07285383c695b0"
"awaking"
"stretch"
"HasSubevent"
"stretch"
"Something you might do while awaking is [MASK] like a cat."
""
"240facd31336ba033330d48f614e2b03"
"awaking"
"yawn"
"HasSubevent"
"yawn"
"Something you might do while awaking is [MASK]."
""
"c7914091e7793d478ea69509b0c708ad"
"bathe"
"shampoo"
"HasSubevent"
"shampoo"
"You can use [MASK] to bathe your dog."
""
"cb580aee35f1199e00adf55e33ca18cc"
"bathing"
"drown"
"HasSubevent"
"drown"
"Something you might do while bathing is [MASK]."
""
"6348412aa11d8584face9495e381389f"
"bathing"
"read"
"HasSubevent"
"read"
"While bathing some people [MASK] magazines."
""
"e461d079140df8332cba125c8e9f894d"
"bathing"
"relax"
"HasSubevent"
"relax"
"Something you might do while bathing is [MASK]."
""
"552e978475f48a14ec9676ec49496d05"
"bathing"
"soak"
"HasSubevent"
"soak"
"Something you need to do before you [MASK] in a hotspring is put on a bathing suit."
""
"d92396a0eb4491fde5f8a0816e6a1fec"
"breathe"
"inhaling"
"HasSubevent"
"inhaling"
"The statement "If you want to breathe then you should inhale and exhale" is true because breathing is taking in oxygen by [MASK] air into your lungs, which sends the oxygen into your bloodstream, and takes the carbon dioxide out of your bloodstream, which you then exhale back into space."
""
"f9acca0e66078c069dad1ca00ea04b2d"
"breathe"
"wheeze"
"HasSubevent"
"wheeze"
"Something that might happen when you breathe is a [MASK]."
""
"22869467491e3405cf8f9ea4bf5511a3"
"breathing"
"choke"
"HasSubevent"
"choke"
"Something you might do while breathing is [MASK]."
""
"2f4d8dcf9eaabf2b9700786e1457b66f"
"breathing"
"cough"
"HasSubevent"
"cough"
"Something that might happen while breathing is [MASK]."
""
"785f45e3c723a6c354c14dc1ea92e3d6"
"breathing"
"hiccup"
"HasSubevent"
"hiccup"
"Something that might happen while breathing is [MASK]."
""
"516f74e91db87c7eb699a5649c910f98"
"breathing"
"inhale"
"HasSubevent"
"inhale"
"The statement "If you want to breathe then you should [MASK] and exhale" is true because breathing is taking in oxygen by inhaling air into your lungs, which sends the oxygen into your bloodstream, and takes the carbon dioxide out of your bloodstream, which you then exhale back into space."
""
"0969817659ab343f5d80ead6ac11eaae"
"breathing"
"live"
"HasSubevent"
"live"
"If you're breathing and have a place to [MASK], everything else is a bonus."
""
"1aa258f1b258a5f886dbaff9278f5f89"
"breathing"
"sigh"
"HasSubevent"
"sigh"
"Something you might do while breathing is [MASK]."
""
"b0b81f4330cc86d2ff409250a66afe5f"
"breathing"
"yawn"
"HasSubevent"
"yawn"
"Something you might do while breathing is [MASK]."
""
"f9214ab1e327da66fad6a5e743a5649f"
"buying"
"act"
"HasSubevent"
"act"
"When I want to buy something, I get into the [MASK] of buying."
""
"34a4fbadffad7ae745ea58b94d8043f8"
"buying"
"bargain"
"HasSubevent"
"bargain"
"The fact "a salesman can offer a good deal" is illustrated with the story:1. a good deal is the right object at the right price2. a good deal is buying a pizza and getting another one free.3. a good deal is a nice car for $1000.004. salesmen get paid to sell things to people like you and me5. a salesman can offer you a good deal, or you may be able to [MASK] with him to lower the price."
""
"70271177b20d1d2e8be99289452231d6"
"buying"
"decide"
"HasSubevent"
"decide"
"To understand the event "Fred was thinking of buying a motorcycle.", it is important to know that fred was also trying to [MASK] wether he was going to buy the motorcycle or not."
""
"a44291626d43d0dee6ad544ea63e7f0a"
"buying"
"feel"
"HasSubevent"
"feel"
"Something you might do while buying fresh fruits and vegetables is [MASK] them to see if they are ripe."
""
"4498bccaf9ea8ef6a34c2eea54b26045"
"buying"
"incisive"
"HasSubevent"
"incisive"
"Something you might do while buying something is be [MASK]."
""
"fb0f146e54e2c6740fc746a2e0a149bb"
"buying"
"insure"
"HasSubevent"
"insure"
"Something you might do while buying something is [MASK] it."
""
"46970fcabc30ed3eed0c5cf587832f59"
"buying"
"investigate"
"HasSubevent"
"investigate"
"Something you might do while buying something is [MASK] it."
""
"ea780665cc8f6fd81a7abc64e202c895"
"buying"
"negotiate"
"HasSubevent"
"negotiate"
"Something that might happen while buying a house is that you will need to [MASK] on the price with the people who are selling the house."
""
"505bea204849d26444eb58d4ec790ad1"
"buying"
"observe"
"HasSubevent"
"observe"
"Something you might do while buying something is [MASK] it."
""
"10df77beb4c8567142335005b182ae4a"
"buying"
"pinch"
"HasSubevent"
"pinch"
"Something you might do while buying something is [MASK] it."
""
"47836ecce2241dd9719d4ea3b64cb207"
"buying"
"ponder"
"HasSubevent"
"ponder"
"Something you might do while buying something is [MASK]."
""
"bab779479ec367ea7b241fc657957bd8"
"buying"
"sample"
"HasSubevent"
"sample"
"Something you might do while buying something is taste a [MASK]."
""
"1875512af960d0b84d819a1c21586900"
"buying"
"selective"
"HasSubevent"
"selective"
"Something you might do while buying something is be [MASK]."
""
"374fafdfbd475e7efda424eac824f1c5"
"buying"
"sharp"
"HasSubevent"
"sharp"
"Something you might do while buying something is be [MASK]."
""
"d8546f11e79de705c8ce334361e17b5c"
"buying"
"skeptical"
"HasSubevent"
"skeptical"
"Something you might do while buying something is be [MASK]."
""
"0786ea8c27c40acf24749550b2e2ad7a"
"buying"
"smell"
"HasSubevent"
"smell"
"Something you might do while buying something is [MASK] it."
""
"c521c524519505b6f287f8eba5d0139a"
"buying"
"study"
"HasSubevent"
"study"
"Something you might do while buying something is [MASK] it."
""
"fdc270bb3b93d04f446cf4be8245b601"
"buying"
"suspicious"
"HasSubevent"
"suspicious"
"The statement "The story "Buying Something" has the step "She talked to me in a soothing tone."" is true because She needed to convince me not to be [MASK] of her."
""
"3d1c4fa4e8490699b0a813a0752dce76"
"buying"
"worry"
"HasSubevent"
"worry"
"Something you might do while buying something is [MASK]."
""
"cce8a8cf6841e300b4c5870ef1eaf832"
"celebrating"
"applause"
"HasSubevent"
"applause"
"Something that might happen while celebrating is [MASK]."
""
"e9a4bf3228ee7603c3109d8500f92e89"
"celebrating"
"boasting"
"HasSubevent"
"boasting"
"Something that might happen while celebrating is [MASK]."
""
"69f3d8b1b3730067a4b8aecbe975317b"
"celebrating"
"bonfire"
"HasSubevent"
"bonfire"
"Another way to say "lighting a fire is for celebrating." is "Some celebrations require the lighting of a [MASK]."."
""
"4bef8830c468de46053952cd25112242"
"celebrating"
"buffet"
"HasSubevent"
"buffet"
"Something that might happen while celebrating is a [MASK]."
""
"607d5a3e4a2df776d35fae176289c44e"
"celebrating"
"chaos"
"HasSubevent"
"chaos"
"Something that might happen while celebrating is [MASK]."
""
"4a4ea547b6440531be2d6a5edd74c3a7"
"celebrating"
"christmas"
"HasSubevent"
"Christmas"
"Something that might happen while celebrating is [MASK]."
""
"24cdc861450515ce8aca02b4f57ebc91"
"celebrating"
"commemoration"
"HasSubevent"
"commemoration"
"Something that might happen while celebrating is a [MASK]."
""
"7d0a4535783a6132c4e07006d57e4018"
"celebrating"
"contest"
"HasSubevent"
"contest"
"Something that might happen while celebrating is a [MASK]."
""
"6f26193105efa5e2e2f8de1d679ea961"
"celebrating"
"dance"
"HasSubevent"
"dance"
"Something you might do while celebrating is [MASK]."
""
"d7413c0d2ab8ade5e641ec8ac53252fa"
"celebrating"
"decorate"
"HasSubevent"
"decorate"
"Something that might happen while celebrating is [MASK]."
""
"0d61f8beec6326cf7a25b88c4945d3a5"
"celebrating"
"drink"
"HasSubevent"
"drink"
"Something you might do while celebrating is [MASK]."
""
"7e0312055c47d2bda78b98a7b0f946e9"
"celebrating"
"exulting"
"HasSubevent"
"exulting"
"Something that might happen while celebrating is [MASK]."
""
"7e919f9acaffd2440c9104139a7980e4"
"celebrating"
"fireworks"
"HasSubevent"
"fireworks"
"Something that might happen while celebrating is [MASK]."
""
"ec83c4d3cb0dfe9ce45be97656f51a12"
"celebrating"
"happy"
"HasSubevent"
"happy"
"The statement "You would celebrate because you want to feel [MASK]" is true because because celebrating is how you lift your spirit."
""
"795010834df1b11db8ce6f8d661e24cc"
"celebrating"
"hosting"
"HasSubevent"
"hosting"
"The statement "Another way to say "having a party is for celebrating" is "People celebrate events by [MASK] parties."" is true because The first phrase in quotes has a very similar meaning to the second phrase in quotes."
""
"683aa19d97efd20d94b7d95ec9151238"
"celebrating"
"hugs"
"HasSubevent"
"hugs"
"Something that might happen while celebrating is [MASK]."
""
"48ee90b957039f5e0772c303433be326"
"celebrating"
"humor"
"HasSubevent"
"humor"
"Something that might happen while celebrating is [MASK]."
""
"bbf44fccbb904a4eebfe04b7d998b36c"
"celebrating"
"indigestion"
"HasSubevent"
"indigestion"
"Something that might happen while celebrating is [MASK]."
""
"672b460ba8b8f75fcb90399cc44b3a5a"
"celebrating"
"kissing"
"HasSubevent"
"kissing"
"Something that might happen while celebrating is [MASK]."
""
"de217b2de766c74ed82633066406ddd8"
"celebrating"
"laughter"
"HasSubevent"
"laughter"
"Something that might happen while celebrating is [MASK]."
""
"caaca7362497755a5f43ee1c5d3c564f"
"celebrating"
"looting"
"HasSubevent"
"looting"
"Something that might happen while celebrating is [MASK]."
""
"0cd4969014d091f01e4e185c1a92bd1a"
"celebrating"
"memorializing"
"HasSubevent"
"memorializing"
"Something that might happen while celebrating is [MASK]."
""
"bf8122c4791aac7e322f9c9f408a18e6"
"celebrating"
"noise"
"HasSubevent"
"noise"
"Something that might happen while celebrating is [MASK]."
""
"014277ee68c8c4eca7ef148b7222b384"
"celebrating"
"parade"
"HasSubevent"
"parade"
"Something that might happen while celebrating is a [MASK]."
""
"c8460d0399052a6650a58753d3faefe0"
"celebrating"
"revelry"
"HasSubevent"
"revelry"
"Something that might happen while celebrating is [MASK]."
""
"44012d700eefdf014b4a7fcddbd44f47"
"celebrating"
"riot"
"HasSubevent"
"riot"
"Something that might happen while celebrating is a [MASK]."
""
"9fd5598dde0afef19e9321a83be9ac82"
"celebrating"
"roasted"
"HasSubevent"
"roasted"
"Something that might happen while celebrating is someone is [MASK]."
""
"6600dd9497a28b7bcbe69d4ae48f5e06"
"celebrating"
"screaming"
"HasSubevent"
"screaming"
"Something that might happen while celebrating is [MASK]."
""
"869739800b7ae036d691922b7d2b6e25"
"celebrating"
"toasted"
"HasSubevent"
"toasted"
"Something that might happen while celebrating is someone is [MASK]."
""
"723c12656f02def08fa855a7e3fdf0ef"
"choke"
"gag"
"HasSubevent"
"gag"
"One of the things you do when you choke is [MASK]."
""
"8b02b810f89e4fdde3fb56dd826ca76c"
"clean"
"neatness"
"HasSubevent"
"neatness"
"Clean is [MASK]."
"Clean is not [MASK]."
"82f179752c1842c625920cedbba7c3aa"
"clean"
"vaccuum"
"HasSubevent"
"vaccuum"
"If you want to clean your room then you should have a [MASK]."
""
"2968f8121e4bec209684ee35827d0d57"
"cleaning"
"declutter"
"HasSubevent"
"declutter"
"Something you might do while cleaning is [MASK]."
""
"6fd30ee4b110dbc0bb322c672c894ece"
"cleaning"
"dust"
"HasSubevent"
"dust"
"Something that might happen while cleaning the house is plugging in the vacuum and pushing it around on the carpet to pick up [MASK], dirt, dog hair, and other small items."
""
"a3edffc34d82ca0dda72b93445af7b3f"
"cleaning"
"dusting"
"HasSubevent"
"dusting"
"Something that might happen while cleaning the house is [MASK]."
""
"2b34465b6c407da23b5feb60d5281e37"
"cleaning"
"sneezing"
"HasSubevent"
"sneezing"
"Something that might happen as a consequence of cleaning your room is not [MASK] much."
""
"35e109a47f487e3f1239f286f333e1f4"
"cleaning"
"sweep"
"HasSubevent"
"sweep"
"Something you might do while cleaning is [MASK] floors."
""
"eb94356f50683fae18ecfee92eec46e8"
"cleaning"
"thinking"
"HasSubevent"
"thinking"
"Something you might do while cleaning is [MASK]."
""
"5263e1fc0d77804d0d0d1e6495e4121a"
"cleaning"
"vacuum"
"HasSubevent"
"vacuum"
"Something that might happen while cleaning the house is plugging in the [MASK] and pushing it around on the carpet to pick up dust, dirt, dog hair, and other small items."
""
"bec8fa18d367cd24ee0a8942347b56e9"
"climb"
"fall"
"HasSubevent"
"fall"
"To understand the event "Sam went to climb a tree.", it is important to know that Sam could possibly [MASK] and break something."
""
"b36800d184995cf36c3ba71d013faf5f"
"climb"
"falling"
"HasSubevent"
"falling"
"Something that might happen when you climb is [MASK]."
""
"79cc175bc215764013d915814dbb5148"
"climb"
"grab"
"HasSubevent"
"grab"
"One of the things you do when you climb is [MASK] something."
""
"0bc3e9ffcf88d07d030a2ffb7ba5e76f"
"cogitate"
"ruminate"
"HasSubevent"
"ruminate"
"One of the things you do when you cogitate is [MASK]."
""
"2ffd172c55f94fbbc3abfd28e0854c44"
"cogitating"
"creativity"
"HasSubevent"
"creativity"
"Something that might happen while cogitating is [MASK]."
""
"55714c46dd3aafbcf351cc8374cefedf"
"cogitating"
"enjoy"
"HasSubevent"
"enjoy"
"Something that might happen while cogitating is you [MASK] it."
""
"7ea00f788be0360c08198b4a1aa1d04f"
"college"
"learn"
"HasSubevent"
"learn"
"There a hundreds of drinking games to [MASK] at college."
""
"450df151bc384dd8b019d866d1bef8c3"
"comfort"
"hug"
"HasSubevent"
"hug"
"The first thing you do when you comfort a friend is give your friend a [MASK]."
""
"7e105e9a555fa7bfe9d62a598bf12a46"
"communicate"
"laughing"
"HasSubevent"
"laughing"
"Something that might happen when you communicate is [MASK]."
""
"570b80abf22ce0611aa77d2c6873c28a"
"communicate"
"misunderstandings"
"HasSubevent"
"misunderstandings"
"Something that might happen when you communicate is [MASK]."
""
"35a454bb3cb44cc2e06b8242c2c063f7"
"communicating"
"distortion"
"HasSubevent"
"distortion"
"Something that might happen while communicating is [MASK]."
""
"b8394f1322422f62905eac01e55be5f1"
"communicating"
"flirting"
"HasSubevent"
"flirting"
"Something you might do while communicating is [MASK]."
""
"65811dc98535cac415053f24ea32eff6"
"communicating"
"knowledge"
"HasSubevent"
"knowledge"
"Communicating is for gaining [MASK]."
""
"4fe2d09d9cded0afafcb9c21706fa4ac"
"communicating"
"learn"
"HasSubevent"
"learn"
"Something that might happen while communicating is you [MASK]."
""

Dataset Card for LAMA: LAnguage Model Analysis - a dataset for probing and analyzing the factual and commonsense knowledge contained in pretrained language models.

@inproceedings{petroni2020how, title={How Context Affects Language Models' Factual Predictions}, author={Fabio Petroni and Patrick Lewis and Aleksandra Piktus and Tim Rockt{"a}schel and Yuxiang Wu and Alexander H. Miller and Sebastian Riedel}, booktitle={Automated Knowledge Base Construction}, year={2020}, url={https://openreview.net/forum?id=025X0zPfn} }

Dataset Summary

This dataset provides the data for LAMA. The dataset include a subset of Google_RE (https://code.google.com/archive/p/relation-extraction-corpus/), TRex (subset of wikidata triples), Conceptnet (https://github.com/commonsense/conceptnet5/wiki) and Squad. There are configs for each of "google_re", "trex", "conceptnet" and "squad", respectively.

The dataset includes some cleanup, and addition of a masked sentence and associated answers for the [MASK] token. The accuracy in predicting the [MASK] token shows how well the language model knows facts and common sense information. The [MASK] tokens are only for the "object" slots.

This version of the dataset includes "negated" sentences as well as the masked sentence. Also, certain of the config includes "template" and "template_negated" fields of the form "[X] some text [Y]", where [X] and [Y] are the subject and object slots respectively of certain relations.

See the paper for more details. For more information, also see: https://github.com/facebookresearch/LAMA

Languages

en

Dataset Structure

Data Instances

The trex config has the following fields:

{'description': 'the item (an institution, law, public office ...) or statement belongs to or has power over or applies to the value (a territorial jurisdiction: a country, state, municipality, ...)', 'label': 'applies to jurisdiction', 'masked_sentence': 'It is known as a principality as it is a monarchy headed by two Co-Princes – the Spanish/Roman Catholic Bishop of Urgell and the President of [MASK].', 'obj_label': 'France', 'obj_surface': 'France', 'obj_uri': 'Q142', 'predicate_id': 'P1001', 'sub_label': 'president of the French Republic', 'sub_surface': 'President', 'sub_uri': 'Q191954', 'template': '[X] is a legal term in [Y] .', 'template_negated': '[X] is not a legal term in [Y] .', 'type': 'N-M', 'uuid': '3fe3d4da-9df9-45ba-8109-784ce5fba38a'}

The conceptnet config has the following fields:

{'masked_sentence': 'One of the things you do when you are alive is [MASK].', 'negated': '', 'obj': 'think', 'obj_label': 'think', 'pred': 'HasSubevent', 'sub': 'alive', 'uuid': 'd4f11631dde8a43beda613ec845ff7d1'}

The squad config has the following fields:

{'id': '56be4db0acb8001400a502f0_0', 'masked_sentence': 'To emphasize the 50th anniversary of the Super Bowl the [MASK] color was used.', 'negated': "['To emphasize the 50th anniversary of the Super Bowl the [MASK] color was not used.']", 'obj_label': 'gold', 'sub_label': 'Squad'}

The google_re config has the following fields:

{'evidences': '[{\'url\': \'http://en.wikipedia.org/wiki/Peter_F._Martin\', \'snippet\': "Peter F. Martin (born 1941) is an American politician who is a Democratic member of the Rhode Island House of Representatives. He has represented the 75th District Newport since 6 January 2009. He is currently serves on the House Committees on Judiciary, Municipal Government, and Veteran\'s Affairs. During his first term of office he served on the House Committees on Small Business and Separation of Powers & Government Oversight. In August 2010, Representative Martin was appointed as a Commissioner on the Atlantic States Marine Fisheries Commission", \'considered_sentences\': [\'Peter F Martin (born 1941) is an American politician who is a Democratic member of the Rhode Island House of Representatives .\']}]', 'judgments': "[{'rater': '18349444711114572460', 'judgment': 'yes'}, {'rater': '17595829233063766365', 'judgment': 'yes'}, {'rater': '4593294093459651288', 'judgment': 'yes'}, {'rater': '7387074196865291426', 'judgment': 'yes'}, {'rater': '17154471385681223613', 'judgment': 'yes'}]", 'masked_sentence': 'Peter F Martin (born [MASK]) is an American politician who is a Democratic member of the Rhode Island House of Representatives .', 'obj': '1941', 'obj_aliases': '[]', 'obj_label': '1941', 'obj_w': 'None', 'pred': '/people/person/date_of_birth', 'sub': '/m/09gb0bw', 'sub_aliases': '[]', 'sub_label': 'Peter F. Martin', 'sub_w': 'None', 'template': '[X] (born [Y]).', 'template_negated': '[X] (not born [Y]).', 'uuid': '18af2dac-21d3-4c42-aff5-c247f245e203'}

Data Fields

The trex config has the following fields:

  • uuid: the id
  • obj_uri: a uri for the object slot
  • obj_label: a label for the object slot
  • sub_uri: a uri for the subject slot
  • sub_label: a label for the subject slot
  • predicate_id: the predicate/relationship
  • sub_surface: the surface text for the subject
  • obj_surface: The surface text for the object. This is the word that should be predicted by the [MASK] token.
  • masked_sentence: The masked sentence used to probe, with the object word replaced with [MASK]
  • template: A pattern of text for extracting the relationship, object and subject of the form "[X] some text [Y]", where [X] and [Y] are the subject and object slots respectively. template may be missing and replaced with an empty string.
  • template_negated: Same as above, except the [Y] is not the object. template_negated may be missing and replaced with empty strings.
  • label: the label for the relationship/predicate. label may be missing and replaced with an empty string.
  • description': a description of the relationship/predicate. description may be missing and replaced with an empty string.
  • type: a type id for the relationship/predicate. type may be missing and replaced with an empty string.

The conceptnet config has the following fields:

  • uuid: the id
  • sub: the subject. subj may be missing and replaced with an empty string.
  • obj: the object to be predicted. obj may be missing and replaced with an empty string.
  • pred: the predicate/relationship
  • obj_label: the object label
  • masked_sentence: The masked sentence used to probe, with the object word replaced with [MASK]
  • negated: same as above, except [MASK] is replaced by something that is not the object word. negated may be missing and replaced with empty strings.

The squad config has the following fields:

  • id: the id
  • sub_label: the subject label
  • obj_label: the object label that is being predicted
  • masked_sentence: The masked sentence used to probe, with the object word replaced with [MASK]
  • negated: same as above, except [MASK] is replaced by something that is not the object word. negated may be missing and replaced with empty strings.

The google_re config has the following fields:

  • uuid: the id
  • pred: the predicate
  • sub: the subject. subj may be missing and replaced with an empty string.
  • obj: the object. obj may be missing and replaced with an empty string.
  • evidences: flattened json string that provides evidence for predicate. parse this json string to get more 'snippet' information.
  • judgments: data about judgments
  • sub_q: unknown
  • sub_label: label for the subject
  • sub_aliases: unknown
  • obj_w: unknown
  • obj_label: label for the object
  • obj_aliases: unknown
  • masked_sentence: The masked sentence used to probe, with the object word replaced with [MASK]
  • template: A pattern of text for extracting the relationship, object and subject of the form "[X] some text [Y]", where [X] and [Y] are the subject and object slots respectively.
  • template_negated: Same as above, except the [Y] is not the object.

Data Splits

There are no data splits.

Dataset Creation

Curation Rationale

This dataset was gathered and created to probe what language models understand.

Source Data

Initial Data Collection and Normalization

See the reaserch paper and website for more detail. The dataset was created gathered from various other datasets with cleanups for probing.

Who are the source language producers?

The LAMA authors and the original authors of the various configs.

Annotations

Annotation process

Human annotations under the original datasets (conceptnet), and various machine annotations.

Who are the annotators?

Human annotations and machine annotations.

Personal and Sensitive Information

Unkown, but likely names of famous people.

Considerations for Using the Data

Social Impact of Dataset

The goal for the work is to probe the understanding of language models.

Discussion of Biases

Since the data is from human annotators, there is likely to be baises.

[More Information Needed]

Other Known Limitations

The original documentation for the datafields are limited.

Additional Information

Dataset Curators

The authors of LAMA at Facebook and the authors of the original datasets.

Licensing Information

The Creative Commons Attribution-Noncommercial 4.0 International License. see https://github.com/facebookresearch/LAMA/blob/master/LICENSE

Citation Information

@inproceedings{petroni2019language, title={Language Models as Knowledge Bases?}, author={F. Petroni, T. Rockt{"{a}}schel, A. H. Miller, P. Lewis, A. Bakhtin, Y. Wu and S. Riedel}, booktitle={In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019}, year={2019} }

@inproceedings{petroni2020how, title={How Context Affects Language Models' Factual Predictions}, author={Fabio Petroni and Patrick Lewis and Aleksandra Piktus and Tim Rockt{"a}schel and Yuxiang Wu and Alexander H. Miller and Sebastian Riedel}, booktitle={Automated Knowledge Base Construction}, year={2020}, url={https://openreview.net/forum?id=025X0zPfn} }

Contributions

Thanks to @ontocord for adding this dataset.

Downloads last month
3,072