summarization_model
string
cnndm_id
string
summary
string
label
string
"T5-11B"
"e135a5c627501cd817dd0f54bbf61e86ba9336ff"
"Barcelona's Champions League hopes were kept alive as they came from behind to beat Paris St-Germain and finish top of Group F."
"1"
"T5-3B"
"a4b473e2dd4770ccea0169d72e1bb482cfda1e20"
"If you're on a budget, here are some tips for saving money in one of the world's most expensive cities, Oslo."
"1"
"T5-Small"
"c347f533ae4864f0a79e340abd53bdc12600790e"
"The number of people who have been wrongly paid tax in the last 12 months has increased by more than a quarter, HM Revenue and Customs has said."
"0"
"T5-Large"
"4f515f307bdcf4eac1acd2ecd464a0e92aa1baaa"
"What do your Premier League clubs need to end the season on a high?"
"1"
"T5-3B"
"d65bbce603133a022c9cc148859f09f90d018fbe"
"Ian Watkins, a paedophile rock star jailed for 29 years, will earn more than £100,000 in royalties while serving a 29-year jail sentence for child sex offences, it has emerged."
"1"
"T5-3B"
"e4dfbfb2d3522bdc021af394454c1783163888dd"
"As the world mourns the death of Margaret Thatcher, the BBC's Damian Barr reports from London on the events of the day."
"0"
"T5-Large"
"b410c8ee81d3c0d3e1cef73ebc35e13a60022b07"
"A man has been charged with the murder of a mother-of-two whose body was found by her 11-year-old son."
"1"
"T5-3B"
"97d984728b2e2775bfce39eda745b0a9a1553ae8"
""I was told I had two years to live, but I met Professor Justin Stebbing and he gave me back my life."
"1"
"T5-3B"
"366138c7193fa4a468bf264cde31ecc0137b95d7"
"A dashboard camera has helped a woman get an $88 parking fine she wasn't entitled to, thanks to footage from her car's camera."
"1"
"T5-3B"
"c90b6789d9e50f9a8446078042cd37fd176ffcf2"
"They were thrilled when they both fell pregnant at around the same time - but never imagined that their double deliveries would spark the ultimate family celebration."
"1"
"T5-11B"
"58d2d9f16edd6fe40100100d9dc3c4e839962588"
"Ireland's Jonny Sexton suffered a black eye after a bruising Six Nations match against France."
"1"
"T5-Small"
"02800bbc00f7b779c7510ce45d72cdc4401071f7"
"The White House has been urged to press ahead with a short-term debt limit fix, but it's not clear how it would work."
"0"
"T5-11B"
"d8140eae8609faaa3b56ea0dd3cfcc7f1011153b"
"Labour's Tristram Hunt has been accused of hypocrisy after revealing he takes lessons himself while condemning plans to allow more unqualified teachers into schools."
"1"
"T5-3B"
"d3bc76fa2aa6350d10e7fefcdd593137e88b88bb"
"A BBC employee who sent a package that was mistaken for a bomb has been suspended over the blunder."
"0"
"T5-Small"
"1ee83856b88d36f2209b300019a5d46ca55b6bdc"
"Liverpool striker Mario Balotelli is a man who is a man of character."
"1"
"T5-11B"
"072c913df706e5c822c2aa8545ad6dcd500a6712"
"A man who served 39 years in prison for the rape and murder of a young girl has been freed after a judge ruled he was given a fair trial."
"0"
"T5-Large"
"f726cd2d59546685e9331373125ad65e90a5d907"
"A van driver who killed a man in a road rage attack has been warned he faces a lengthy jail term."
"1"
"T5-Large"
"99331f20ba5cde9d28cd3bd096f3f7e849629d8a"
"The family of a boy shot twice outside his home in Milwaukee has decided to leave the area and will not stay for another night."
"1"
"T5-Large"
"671eb41b1b5b8ebf17c4a5953266661d8a523907"
"A former chef of Kim Jong Un's father has described his reunion with the young leader, saying he was "betrayed" by Kim."
"0"
"T5-3B"
"da53f799eafca26f36e306364dc5b9e6a4c21e52"
"""Dancing With the Stars"" champ Ian Ziering is reportedly swapping the ballroom for a striptease with the Chippendales."
"1"
"T5-Small"
"3f671415fbec654756ad80211821e4c42545f416"
"Iraq's government has said it will revise its plan to increase production in the coming years, despite the recent violence in the country."
"1"
"T5-Base"
"a32af72f365970a5cac8cb0c4ba8c00c1ace6ff9"
"A group of women in Uganda is tackling the issue of energy poverty by empowering women to earn a steady income."
"1"
"T5-Large"
"72a555c33d8ecd4d7586f2f6cda9bb995c9077ec"
"China is behind a massive cyber-espionage campaign against the United States, a leading cybersecurity expert has told CNN."
"1"
"T5-3B"
"f3f60734fe2e5c5ba168e5908bd5d5bc73d98d1f"
"A woman who shattered a bridesmaid's face with a wine glass at a wedding party has been jailed for four and a half years."
"1"
"T5-11B"
"c5013d5ffb32420be3643f88729052ca7f870dd6"
"Amid the escalating violence in Iraq and Syria, the BBC's Scott Shane looks at the latest developments in the conflict."
"0"
"T5-Large"
"18ab7a561b479b38e20c9bb52ee675d339de215a"
"The US senator from Arizona has defended the use of torture in the war on terrorism, saying that the tactics were ""not just ineffective but also damaging.""
"0"
"T5-3B"
"870573203dab1515db0edb617c5774fe9529501c"
"A charter school in New Orleans is revising a policy that required pregnant students to be removed from class and home-schooled, a school official says."
"0"
"T5-11B"
"bec88a7292476903d5df86377ea9c45a56274bfb"
"A former felon who helped launch a US presidential campaign is helping a Republican build a bridge to African-American voters."
"0"
"T5-Small"
"6aeb270408c5e64babb60793a92c76c46252ba37"
"The police investigating the death of a mother of two in custody have been removed from their duties."
"1"
"T5-11B"
"f5aff3d6234eae416ebc0b153aee858f1e58d2cf"
"The UK Ministry of Defence (MoD) has said it is investigating reports that a British-trained Iraqi policeman may have saved the lives of dozens of Shia pilgrims by throwing himself on a suicide bomber."
"0"
"T5-Base"
"776b2d775e8e661f4592ad39cf314530d4b9507f"
"A US TV show host has been accused of putting filming of his latest documentary on a house he bought after a mother and her three children were possessed."
"1"
"T5-Small"
"d9184d567834003417acb1fbe8007da5419343f7"
"A bus driver who stole a bus and drove away with no insurance has been jailed for two years."
"0"
"T5-Large"
"41b7b6fc94aad249e483e439187c890adc496c89"
"A shortlist of three candidates has been drawn up for the vacant Fulham manager's job, BBC Radio Solent understands."
"1"
"T5-Small"
"99db081a58a48097e2e6aa2b56890c562750b5fe"
"A Vietnamese woman has been found guilty of severing her husband's penis and slicing off his tofu, a court has heard."
"0"
"T5-Small"
"f2ee3434b505f04a8dff0867bd0a954b4a08c84a"
"The future of Britain’s nuclear deterrent could be at the heart of a deal worth £31bn worth of a deal, the BBC understands."
"1"
"T5-11B"
"66e2092d4ea3a7cbeb504e512c63b9c3ffc3d5b0"
"A 90-year-old Canadian pilot is still flying his beloved Tiger Moth plane - despite the fact he first started flying in it when he was a teenager."
"1"
"T5-Base"
"e1649a343b9ae8607e9462d85e7117da01ecc129"
"The Today Show host is set to run the Boston Marathon in a bid to raise money for victims of the Boston Marathon bombings."
"1"
"T5-Small"
"61fe1d2c34f1f289fbf573f63ce113bdae3b1352"
"The scandal surrounding the mistress of French President Francois Hollande has been uncovered by a BBC investigation."
"0"
"T5-Large"
"ae5103d6cae3b4032414c77e1c4582272cf93b71"
"YouTube users will soon be able to give uploaded videos a rating to help protect children from inappropriate online content, according to the British Board of Film Classification (BBFC)."
"1"
"T5-Large"
"80971fb266f5a3ebcbfc360a55630fe6ae287ad2"
"A couple have been given a supposedly ancient hot cross bun - but it could be the world's oldest - dating back to the year the slave trade was abolished."
"1"
"T5-Large"
"be15aa65f6c29309215de3e2c54b5f8bae341892"
"ITV’s The X Factor has been criticised for a lack of singing during its latest series."
"1"
"T5-Large"
"69c8917177a1736b61ccb67c5f18ad6e2ef8275d"
"Toners are a staple part of many women's beauty regimes - but do they do more harm than good?"
"1"
"T5-11B"
"34af053ed9ede57a027810b656f2b387d91ffdef"
"American Airlines grounded flights across the US on Tuesday due to a technical glitch."
"1"
"T5-Base"
"5400ca8b3cfc1790c1e06cae4f4a3e6cb79dc806"
"Liverpool manager Brendan Rodgers says his side are back to their best after a 'brilliant' win over Tottenham."
"1"
"T5-Large"
"16a9a9c924ef0bc0ada2d02db8f0d856a9da212a"
"DNA samples from a kidnapping suspect in Ohio have been sent to a crime lab in the state to see if they can link him to other crimes."
"1"
"T5-Small"
"8a83c962700c63be6027e339e99a34f5e872694b"
"The Daily Record has learned that Rachel Canning and her ex-boyfriend have dropped domestic violence restraint orders against each other."
"1"
"T5-Small"
"2e29ff8c5c16199aa7dd6ae3b8d72ed5fc7f0283"
"A judge in Guatemala has said the couple's bid to become the country's next president is a "personal right, the human right, the nepotism" of the constitution."
"0"
"T5-3B"
"bc0f194d2c9cf432662da33bf94e58c0100e0d29"
"The CDC has issued a warning to patients and doctors that a deadly outbreak of meningitis linked to a tainted batch of steroids could spread rapidly."
"1"
"T5-11B"
"9ca7ce00bb2c2ab29346a7cf6a4485616d8ac69b"
"Police are searching for a man believed to be linked to the disappearance of a 17-year-old girl in Massachusetts."
"1"
"T5-3B"
"393375a6fd38eb0da4323b004dbc66888820f3ac"
"Pork DNA has been found in Halal chicken sausages served in a Westminster school, the council has confirmed."
"1"
"T5-Base"
"5b82fdfbf169eb5de6f30485e44c28b7af5f26dd"
"Pope Francis has been sworn in as the new Pope of Argentina, a move feared to be a catalyst for nationalism."
"0"
"T5-Small"
"51ff6ebe9ab156c07506d02493963bb68a3de0d6"
"A teenager who collapsed and almost died after drinking energy drinks has told of her experience on a BBC show."
"0"
"T5-3B"
"149c646a3a824b75f75735c484e97dc561b31dad"
"North Korea has destroyed a key part of its nuclear weapons program in a symbolic move, a CNN correspondent reports."
"1"
"T5-Large"
"6c2c6826a83145ffd45eb59c02943475882d5abb"
"The annual run of the Pamplona bulls in Spain has been a highlight of the city's festivities for centuries, but it has rarely gone as planned."
"0"
"T5-Small"
"7d7304f54e5643be1148944c47c3917b49c9ce5f"
"A man has been charged with trespassing after a tiger jumped out of a car and rolled under a wire, prompting a police investigation."
"0"
"T5-Base"
"be61eb82073a379fa779003ff45640787d0091e5"
"The White House has reacted angrily to President Vladimir Putin's comments that Russia is "stumbling into a revisionist Cold War mindset"."
"0"
"T5-Base"
"042cd99976d91af95e6b065befe2308ab78bd875"
"The legal services commission has blocked a request for information about how much taxpayers’ money they spent on legal aid during the Stephen Lawrence murder trial."
"1"
"T5-Base"
"3a1c0448c239dfdbc1398f17c3a626e493206285"
"A 19-year-old woman who has spent £2,000 on her outfits has revealed her new style."
"1"
"T5-Large"
"5d919e9be603dfd26deef78abda4d84c682a8d2b"
"The deaths of a baby girl and her father have sparked a bitter custody battle between the parents."
"0"
"T5-3B"
"de0f09797724610c8b23e6c94cdb713d8e952d66"
"The number of westerners joining the fight in Syria is ""unprecedented,'' a senior U.S. official said."
"1"
"T5-Base"
"cb4e8170bcf56cc76a3b2a41924a75afb27eb54d"
"It's a bit of a prank, but it's not a sexist joke."
"0"
"T5-Small"
"caba1ef1af497fa5a9dc864a8ff441c8cfbf9869"
"The NFL's most-wanted player has been charged with driving under the influence of alcohol."
"0"
"T5-3B"
"8fa95033d9682438cbc134ac88331b13c010fd9b"
"Manchester United's new signings could be announced as early as Monday, with Ed Woodward insisting the club are still a huge attraction to top players."
"1"
"T5-3B"
"dc58ee6b1761919cc74c8c8f292ea856d19b57b9"
"A pilot involved in the search for the missing Malaysia Airlines jet has become an internet sensation after online observers hailed his chiselled good looks."
"1"
"T5-3B"
"8b81634fb722af8a6943a8a92c6577c5073be723"
"A grieving family have told how they learned the devastating news that their devoted daughter was killed in a head-on crash."
"0"
"T5-11B"
"c4808548c12a8b6b9d8cdb6a722c2761293496b6"
"A man with a massive tumour on his neck has had it removed after years of suffering with the condition."
"1"
"T5-11B"
"0c74e1ae61834335e879ee1731ff523f72939d2b"
"The illegal wildlife trade is a multi-billion dollar industry that is fueling conflict and undermining governments, and it is getting out of control."
"1"
"T5-Base"
"a91aa9263208c74da4bcd50e515625b1bce46e60"
"Sale Sharks fly-half Danny Cipriani has been left out of England's squad for the autumn internationals after being left out of the QBE Series."
"1"
"T5-Large"
"d0993609606029d606047823257cbaaa7d2d8abc"
"The Home Office has been accused of failing to tell Abu Qatada he still had time to appeal against his deportation to Jordan."
"0"
"T5-3B"
"dc5d9dcda8938476d6a056deb128396d29a29a30"
"A dementia sufferer who died while handcuffed to a hospital bed was one of a string of cases at a London immigration centre where the elderly and vulnerable were shackled, a report has found."
"1"
"T5-Large"
"3f92db67eda5b964c92fdc7b724b2ae6a4524fc8"
"Celtic manager says he is confident he can sign a goalkeeper to replace Fraser Forster if the England international is sold."
"1"
"T5-11B"
"6195800abef2e5419b643c05c0e8432586d67abd"
"The cheap, easy-to-get formula that has made China the world's manufacturing base is coming to an end, and the change will affect global businesses and consumers, argues the Financial Times's Paul Carter."
"1"
"T5-Small"
"76e4e7c4cc6273a6fb41bce755946016ba31a2af"
"Manchester City's most important player is the Spaniard's best player by far, according to a former player."
"1"
"T5-11B"
"e4770660c404c2d79aa5d29df301b43a0ed4949e"
"Scientists are working on a project to produce a genetically modified (GM) apple that will not trigger an allergic reaction in those who are sensitive to the fruit."
"1"
"T5-11B"
"63c6bdfe127f252b82bfc4c701c7876998128173"
"Hollywood's biggest names are attempting to crack the Indian film market, but are they finding success?"
"1"
"T5-Small"
"b8136991b9967b6d4ffe6f2ce92e018959667746"
"Lord Justice Fulford, who was appointed as a judge in the wake of the paedophile campaigning scandal, has been suspended from the Court of Appeal."
"0"
"T5-11B"
"aef8ef6d611d6e77e3a70c8e3e18d677f21bc061"
"West Ham manager Slaven Bilic has been given permission to speak to Valencia about their bid for Cheikhou Kouyate."
"0"
"T5-Base"
"ed2701d5f3f55443b55f9b1969bbb6ecbe675e69"
"Everton striker Steven Naismith has donated tickets to unemployed people in Liverpool to help them enjoy a day out at a Premier League match."
"1"
"T5-Large"
"624b504c2733a0dc3c276ec0c1041bcedc6bd95a"
"A decorated RAF pilot who hunted down some of the UK's most deadly spies has died at the age of 98."
"1"
"T5-Large"
"12d11bd91ab0acdecea530e3e1c5efd7c4312358"
"Roger Federer beat Jo-Wilfried Tsonga to reach the Australian Open semis."
"1"
"T5-3B"
"658162f15c255252c8cd724c29bea9821b6146a1"
"A new species of human may have been discovered by scientists studying a bone found in a Siberian cave, according to a report in the journal Nature."
"1"
"T5-3B"
"644c4f6a827414b05cbb6bdde0326bcbbfd5fc7f"
"""Ramadan has not arrived at the ideal moment for a player to play a football match,"" Inter Milan coach Jose Mourinho said after dropping a player from his team."
"1"
"T5-Large"
"5052fee31cdc2a0a4e60c6d9a01f71af1007b328"
"The Advertising Standards Authority (ASA) has found the NHS is misleading the public by failing to list the cost of calling some 0844 GP phone numbers."
"1"
"T5-Base"
"666aee1d3139fb17c4cf5c383370f9e149ff16d8"
"It's that time of year again and it's time for binky the blond to get the bare minimum."
"0"
"T5-11B"
"fb5c3e8abd68e2a5c2e89579fc03c76ab934fcfd"
"A courtroom confrontation that ended with a woman being pinned to the ground and a police officer reaching into her mouth to retrieve a pill is now the subject of a viral video."
"1"
"T5-3B"
"964f461e32dcb6a8d19adbb7c4ee2d8f54e52d7e"
"Elephants are known to be one of the world's most intelligent animals, but now it seems they can also tell the time."
"0"
"T5-Small"
"17662736c0b0056d18a33c078f04b64bc369ab0b"
"The Pentagon has defended its cyberattack strategy after it uncovered the "previously classified" cyberattack."
"1"
"T5-Large"
"ca431d698b8df6cd0188ff8774e2710b4bf528d4"
"Tottenham's new manager is preparing for his first competitive game at White Hart Lane after a 2-0 win over QPR in Cyprus."
"0"
"T5-Small"
"b2b30a38f47d12c228c6186ed04ef47aa3482669"
"A team of tech-savvy inventors have invented the world's first greenhouse."
"1"
"T5-11B"
"1e998102ff832608d25b7c2a966df2b5fc5a3476"
"The world's economic watchdog has warned that a rise in financial instability could threaten global growth."
"1"
"T5-3B"
"0009ebb1967511741629926ef9f5faea2bb6be24"
"Hawaiian Airlines has the best on-time performance of the 14 largest U.S. airlines, according to a new report."
"1"
"T5-11B"
"8d18d837945dd0ca3c9b99cb85cb97f45c37f3e4"
"Roger Federer's Wimbledon triumph has been described as ""the greatest player to have ever played the game"" by one of the men's game's most respected coaches."
"1"
"T5-Small"
"d391164eb789c29456f697bdb049b058ff77acc7"
"Bahrain has condemned the sentencing of protesters who were sentenced to life in prison for allegedly plotting to overthrow the government."
"0"
"T5-Small"
"7688f3ed9e03177aaaec3fcff881a0505b404bb8"
"The Senate's debate on the Koch brothers' campaign spending has been interrupted by a debate about the billionaires' influence on the economy."
"0"
"T5-3B"
"d677052589e58a15610477188b8ee1d1537ea727"
"Connecticut will become the 17th state to abolish the death penalty, the Associated Press reports."
"0"
"T5-Large"
"5c6edb53cdb8e885a7767ac9ffa6fa24c20c35bc"
"A woman who claimed she was wrongly arrested and ill-treated by police over a taxi-harness bust-up has won a landmark High Court battle."
"0"
"T5-11B"
"469f17dd0d48b4ef23a6929c425f0da389af834b"
"A US mother has said she was sober enough to breastfeed her six-month-old baby while having two beers with dinner."
"1"
"T5-Base"
"d259d8b19632fb598c25cec86658ef5c5ed98454"
"A man who lied about falling over in a supermarket has been given a suspended prison sentence."
"1"
"T5-Base"
"9459c25241e099399cff4579e66d00347507832e"
"Ravel Morrison could be sold to QPR for £15m if the club can secure a new deal, reports BBC Radio Manchester."
"0"
"T5-Base"
"6420430469f999ce00977cd6b2f033d4f28a8d19"
""It was like a shock," says the man who led us to the airport in Guinea, where we were greeted by a crowd of people."
"0"

TrueTeacher

Dataset Summary

This is a large-scale synthetic dataset for training Factual Consistency Evaluation models, introduced in the TrueTeacher paper (Gekhman et al, 2023).

Dataset Details

The dataset contains model-generated summaries of articles from the train split of the CNN/DailyMail dataset (Hermann et al., 2015) which are annotated for factual consistency using FLAN-PaLM 540B (Chung et al.,2022). Summaries were generated using summarization models with different capacities, which were created by fine-tuning T5 (Raffel et al., 2020) on the XSum dataset (Narayan et al., 2018). We used the following 5 capacities: T5-11B, T5-3B, T5-large, T5-base and T5-small.

Data format

The data contains json lines with the following keys:

  • "summarization_model" - The summarization model used to generate the summary.
  • "cnndm_id" - The original id from the CNN/DailyMail dataset, this need to be used in order to retrieve the corresponding article from CNN/DailyMail (which was used as the grounding document).
  • "summary" - The model-generated summary.
  • "label" - A binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).

Here is an example of a single data item:

{
  "summarization_model": "T5-11B",
  "cnndm_id": "f72048a23154de8699c307e2f41157abbfcae261",
  "summary": "Children's brains are being damaged by prolonged internet access, a former children's television presenter has warned."
  "label": "1",
}

Loading the dataset

To use the dataset, you need to fetch the relevant documents from the CNN/DailyMail dataset. The follwoing code can be used for that purpose:

from datasets import load_dataset
from tqdm import tqdm

trueteacher_data = load_dataset("google/trueteacher", split='train')
cnn_dailymail_data = load_dataset("cnn_dailymail", version="3.0.0", split='train')
cnn_dailymail_articles_by_id = {example['id']: example['article'] for example in cnn_dailymail_data}
trueteacher_data_with_documents = []
for example in tqdm(trueteacher_data):
  example['document'] = cnn_dailymail_articles_by_id[example['cnndm_id']]
  trueteacher_data_with_documents.append(example)

Intended Use

This dataset is intended for a research use (non-commercial) in English.

The recommended use case is training factual consistency evaluation models for summarization.

Out-of-scope use

Any use cases which violate the cc-by-nc-4.0 license.

Usage in languages other than English.

Citation

If you use this dataset for a research publication, please cite the TrueTeacher paper (using the bibtex entry below), as well as the CNN/DailyMail, XSum, T5 and FLAN papers mentioned above.

@misc{gekhman2023trueteacher,
      title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models}, 
      author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
      year={2023},
      eprint={2305.11171},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
Downloads last month
286
Edit dataset card
Evaluate models HF Leaderboard

Models trained or fine-tuned on google/trueteacher