Abstract

Recent advances in summarization provide models that can generate summaries of higher quality. Such models now exist for a number of summarization tasks, including query-based summarization, dialogue summarization, and multi-document summarization. While such models and tasks are rapidly growing in the research field, it has also become challenging for non-experts to keep track of them. To make summarization methods more accessible to a wider audience, we develop SummerTime by rethinking the summarization task from the perspective of an NLP non-expert. SummerTime is a complete toolkit for text summarization, including various models, datasets and evaluation metrics, for a full spectrum of summarization-related tasks. SummerTime integrates with libraries designed for NLP researchers, and enables users with easy-to-use APIs. With SummerTime, users can locate pipeline solutions and search for the best model with their own data, and visualize the differences, all with a few lines of code. We also provide explanations for models and evaluation metrics to help users understand the model behaviors and select models that best suit their needs. Our library, along with a notebook demo, is available at https://github.com/Yale-LILY/SummerTime.

1 Introduction↩︎

Figure 1: SummerTime is a toolkit for helping non-expert users to find the best summarization models for their own data and use cases.

The goal of text summarization is to generate short and fluent summaries from longer textual sources, while preserving the most salient information in them. Benefiting from recent advances of deep neural networks, in particular sequence to sequence models, with or without attention [1]–[3], current state-of-the-art summarization models produce high quality summaries that can be useful in practice cases [4], [5]. Moreover, neural summarization has broadened its scope with the introduction of more summarization tasks, such as query-based summarization [6], [7], long-document summarization [8], multi-document summarization [9], [10], dialogue summarization [7], [11]. Such summarization tasks can also be from different domains [8], [12], [13].

However, as the field rapidly grows, it is often hard for NLP non-experts to follow all relevant new models, datasets, and evaluation metrics. Moreover, those models and datasets are often from different sources, making it a non-trivial effort for the users to directly compare the performance of such models side-by-side. This makes it hard for them to decide which models to use. The development of libraries such as Transformers [14] alleviate such problems to some extent, but they only cover a narrow range of summarization models and tasks and assume certain proficiency in NLP from the users, thus the target audience is still largely the research community.

To address those challenges for non-expert users and make state-of-the-art summarizers more accessible as a tool, we introduce SummerTime, a text summarization toolkit intended for users with no NLP background. We build this library from this perspective, and provide an integration of different summarization models, datasets and evaluation metrics, all in one place. We allow the users to view a side-by-side comparison of all classic and state-of-the-art summarization models we support, on their own data and combined into pipelines that fit their own task. SummerTime also provides the functionality for automatic model selection, by constructing pipelines for specific tasks first and iteratively evaluation to find the best working solutions. Assuming no background in NLP, we list “pros and cons” for each model, and provide simple explanations for all the evaluation metrics we support. Moreover, we go beyond pure numbers and provide visualization of the performance and output of different models, to facilitate users in making decisions about which models or pipelines to finally adopt.

The purpose of SummerTime is not to replace any previous work, on the contrary, we integrate existing libraries and place them in the same framework. We provide wrappers around such libraries intended for expert users, maintaining the user-friendly and easy-to-use APIs.

2.1 Text Summarization↩︎

Text summarization has been a long-standing task for natural language processing. Early systems for summarization had been focusing on extractive summarization [15], [16], by finding the most salient sentences from source documents. With the advancement of neural networks [1], [2], the task of abstractive summarization has been receiving more attention [17]–[22] while neural-based methods have also been developed for extractive summarization [23]–[28]. Moreover, the field of text summarization has also been broadening into several subcategories, such as multi-document summarization [9], [10], [29], [30], query-based summarization [31]–[37] and dialogue summarization [7], [11], [38]–[41]. The proposed tasks, along with the datasets can also be classified by domain, such as news [10], [12], [42], meetings [7], [43], [44], scientifc literature [8], [45], and medical records [13], [46], [47].

2.2 Existing Systems for Summarization↩︎

Transformers [14] includes a large number of transformer-based models in its Modelhub¹, including BART [5] and Pegasus [4], two strong neural summarizers we also use in SummerTime. It also hosts datasets for various NLP tasks in its Datasets² library [48]. Despite the wide coverage in transformer-based models, Transformers do not natively support models or pipelines that can handle aforementioned subcategories of summarization tasks. Moreover, it assumes certain NLP proficiency in its users, thus is harder for non-expert users to use. We integrate with Transformers and Datasets to import the state-of-the-art models, as well as summarization datasets into SummerTime, under the same easy-to-use framework.

Another library that we integrate with is SummEval [49], which is a collection of evaluation metrics for text summarization. SummerTime adopts a subset of such metrics in SummEval that are more popular and easier to understand. SummerTime also works well with SummVis [50], which provides an interactive way of analysing summarization results on the token-level. We also allow SummerTime to store output in a format that can be directly used by SummVis and its UI.

Other systems also exist for text summarization. MEAD³ is a platform for multi-lingual summarization. Sumy⁴ can produce extractive summaries from HTML pages or plain texts, using several traditional summarization methods including [15] and [16]. OpenNMT⁵ is mostly for machine translation, but it also hosts several summarization models such as [51].

3 SummerTime↩︎

The main purpose of SummerTime is to help non-expert users navigate through various summarization models, datasets and evaluation metrics, and provide simple yet comprehensive information for them to select the models that best suit their needs. 1 shows how SummerTime is split into different modules to help users achieve such goal.

We will describe in detail each component of SummerTime in the following sections. With 3.1, we introduce the models we support in all subcategories of summarization; in 3.2 we list all the existing datasets we support and how users can create their own evaluation set. Finally in 3.3, we explain the evaluation metrics included with SummerTime and how they can help users find the most suitable model for their task.

3.1 Summarization Models↩︎

Here we introduce the summarization tasks SummerTime covers and the models we include to support these tasks. We first introduce the single-document summarization models (i.e., “base models”) in SummerTime, and then we show how those models can be used in a pipeline with other methods to complete more complex tasks such as query-based summarization and multi-document summarization.

Single-document Summarization↩︎

The following base summarization models are used in SummerTime. They all take a single document and generate a short summary.
TextRank [15] is a graph-based ranking model that can be used to perform extractive summarization;
LexRank [16] is also a graph-based extractive summarization model, which is originally developed for multi-document summarization, but can also be applied to a single document. It uses centrality in a graph representation of sentences to measure their relative importance;
BART [5] is an autoencoder model trained with denoising objectives during training. This seq2seq model is constructed with a bidirectional transformer encoder and a left-to-right transformer decoder, which can be fine-tuned to perform abstractive summarization;
Pegasus [4] proposes a new self-supervised pretraining objective for abstractive summarization, by reconstructing the target sentence with the remaining sentences in the document, it also shows strong results in low-resource settings;
Longformer [52] addresses the problem of memory need for self-attention models by using a combination of sliding window attention and global attention to approximate standard self-attention. It is able to support input length of 16K tokens, a large improvement over previous transformer-based models.

Multi-document Summarization↩︎

For multi-document summarization, we adopt two popular single-document summarizers to complete the task, as this is shown to be effective in previous work [10].
Combine-then-summarize is a pipeline method to handle multiple source documents, where the documents are concatenated and then a single document summarizer is used to produce the summary. Note that the length of the combined documents may exceed the input length limit for typical transformer-based models;
Summarize-then-combine first summarizes each source document independently, then merges the resulting summaries. Compared to the combine-then-summarize method, it is not affected by overlong inputs. However, since each document is summarized separately, the final summary may contain redundant information [30].

Query-based Summarization↩︎

For summarization tasks based on queries, we adopt a pipeline method and first use retrieval methods to identify salient sentences or utterances in the original document or dialogue, then generate summaries with a single-document summarization model.
TF-IDF retrieval is used in a pipeline to first retrieve the sentences that are most similar to the query based on the TF-IDF metric;
BM25 retrieval is used in the same pipeline, but BM25 is used as the similarity metric for retrieving the top-\(k\) relevant sentences.

Dialogue Summarization↩︎

Dialogue summarization is used to extract salient information from a dialogue. SummerTime includes two methods for dialogue summarization.
Flatten-then-summarize first flattens the dialogue data while preserving the speaker information, then a summarizer is used to generate the summary. [7] found that this presents a strong baseline for dialogue summarization.
HMNet [41] explores the semantic structure of dialogues and develops a hierarchical architecture to first encode each utterance then aggregate with another encoder in modeling the long dialogue script. It also exploits role vectors to perform better speaker modeling.

Figure 2: A short description of the Pegasus model, SummerTime includes such short descriptions for each supported models to help user making choices.

Since we assume no NLP background of our target users, we provide a short description for every model to illustrate the strengths and weaknesses for each model. Such manually written descriptions are displayed when calling a static get_description() method on the model class. A sample description is shown in 2.

3.2 Datasets↩︎

Table 1: The summarization datasets included in . “Dial.” is short for “Dialogue” while “Lang.” denotes the languages of each of the datasets.
Dataset	Domain	Size	Src. length	Tgt. length	Lang.
CNN/DM(3.0.0)	News	300k	781	56	En
Multi-News	News	56k	2.1k	263.8	En
SAMSum	Open-domain	16k	94	20	En
XSum	News	226k	431	23.3	En
ScisummNet	Scientific articles	1k	4.7k	150	En
QMSum	Meetings	1k	9.0k	69.6	En
ArXiv	Scientific papers	215k	4.9k	220	En
PubMedQA	Biomedial	273.5k	239	43	En
SummScreen	TV shows	26.9k	6.6k	337.4	En
MLSum	News	1.5M	635	31.8
Ru, Tr

With SummerTime, users can easily create or convert their own summarization datasets and evaluate all the supporting models within the framework. However, in the case that no such datasets are available, SummerTime also provides access to a list of existing summarization datasets. This way, users can select models that perform the best on one or more datasets that are similar to their task.

CNN/DM [12] contains news articles from CNN and Daily Mail. Version 1.0.0 of it was originally developed for reading comprehension and abstractive question answering, then the extractive and abstractive summarization annotations were added in version 2.0.0 and 3.0.0, respectively;
Multi-News [10] is a large-scale multi-document summarization dataset which contains news articles from the site newser.com with corresponding human-written summaries. Over 1,500 sites, i.e. news sources, appear as source documents, which is higher than the other common news datasets.
SAMSum [11] is a dataset with chat dialogues corpus, and human-annotated abstractive summarizations. In the SAMSum corpus, each dialogue is written by one person. After collecting all the dialogues, experts write a single summary for each dialogue.
XSum [42] is a news summarization dataset for generating a one-sentence summary aiming to answer the question “What is the article about?”. It consists of real-world articles and corresponding one-sentence summarization from British Broadcasting Corporation (BBC).
ScisummNet [45] is a human-annotated dataset made for citation-aware scientific paper summarization (Scisumm). It contains over 1,000 papers in the ACL anthology network as well as their citation networks and their manually labeled summaries.
QMSum [7] is designed for query-based multi-domain meeting summarization. It collects the meetings from AMI and ICSI dataset, as well as the committee meetings of the Welsh Parliament and Parliament of Canada. Experts manually wrote summaries for each meeting.
ArXiv [8] is a dataset extracted from research papers for abstractive summarization of single, longer-form documents. For each research paper from arxiv.org, its abstract is used as ground-truth summaries.
PubMedQA [53] is a question answering dataset on the biomedical domain. Every QA instance contains a short answer and a long answer, latter of which can also be used for query-based summarization.
SummScreen [38] consists of community contributed transcripts of television show episodes from The TVMegaSite, Inc. (TMS) and ForeverDream (FD). The summary of each transcript is the recap from TMS, or a recap of the FD shows from Wikipedia and TVMaze.
MLSum [54] is a large-scale multilingual summarization dataset. It contains over 1.5M news articles in five languages, namely French, German, Spanish, Russian, and Turkish.

A summary of all datasets included in SummerTime is shown as 1, it is worth noticing that the fields in this table (i.e., domain, query-based, multi-doc, etc) are also incorporated in each of the dataset classes (e.g., SAMSumDataset) as class variables, so that such labels can later be used to identify applicable models. Similar with the models classes, we include a short description for each of the datasets. Note that the datasets, either existing ones or user created are mainly for evaluation purposes. We leave the important task of fine-tuning the models on these datasets for future work, for which we describe in more detail in 5.

3.3 Evaluation Metrics↩︎

To evaluate the performance of each supported model on certain dataset, SummerTime integrates with SummEval [49] and provides the following evaluation metrics for the users to understand model performance:
ROUGE [55] is a recall-oriented method based on overlapping n-grams, word sequences, and word pairs between the generated output and the gold summary;
BLEU [56] measures n-gram precision and employs a penalty for brevity, BLEU is often used as an evaluation metric for machine translation;
ROUGE-WE [57] aims to go beyond surface lexical similarity and uses pretrained word embeddings to measure the similarity between different words and presents a better correlation with human judgements;
METEOR [58] is based on word-to-word matches between generated and reference summaries, it consider two words as “aligned” based on a Porter stemmer [59] or synonyms in WordNet [60];
BERTScore [61] computes token-level similarity between sentences with the contextualized embeddings of each tokens.

Since we assume no NLP background from our target users, we make sure that SummerTime provides a short explanation for each evaluation metric as well as a clarification whether high or low scores are better for a given evaluation metric, to help the non-expert users understand the meaning of the metrics and use them to make decisions.

Figure 3: An illustration of how SummerTime finds solutions to a specific tasks defined by a dataset. The red star denotes that an ending point is reached.

Figure 4: Select(\mathcal{M}, \mathcal{D}, \mathcal{E}) — Figure 4: Select(\(\mathcal{M}, \mathcal{D}, \mathcal{E}\))

4 Model Selection↩︎

In this section, we describe in detail about the workflow of SummerTime and how it can help our non-expert users find the best models for their use cases, which is one of the main functionalities that makes SummerTime stands out from similar libraries. A concrete code example of this is shown in 5.

Figure 5: Example code for using SummerTime. Additionally, we show two ways for performing model selection and evaluation.

4.0.0.1 Create/select datasets

The user would first load a dataset with the APIs we provide. During the process, the users also need to specify some Boolean attributes (e.g., is_query_based, is_dialogue_based) to facilitate next steps. Alternatively, the user can also choose to use one of the datasets that are included in SummerTime, where such attributes are already specified in 1.

4.0.0.2 Construct pipelines

After identifying the potential pipeline modules (e.g., query-based module, dialogue-based module) that are applicable to the task, SummerTime automatically constructs solutions to a specific dataset by combining the pipelines and summarization models specified in 3.1. It further places all such constructed solutions in a pool for further evaluation and selection purposes. An example of this process in shown in 3.

4.0.0.3 Search for the best models

As shown in 3, there can be a large pool of solutions to be evaluated. To save time and resources in searching for best models, SummerTime adopts the idea of successive halving [62], [63]. More specifically, SummerTime first uses a small number of examples from the dataset to evaluate all the candidates and eliminate models that are surpassed by at least one other model on every evaluation metric, then it does so iteratively and gradually increases the evaluation set size to reduce the variance. As shown in 4, the final output is a set of competing models \(M\) that are better⁶ than one another on at least one metric.

4.0.0.4 Visualization

Figure 6: Examples of the visualization SummerTime provides for the users to better compare the performance between different models..

In addition to showing the numerical results as tables, SummerTime also allows the users to visualize the differences between different models with different charts and SummVis [50]. 6 shows some examples of such visualization methods SummerTime provides. A scatter plot can help the users understand the distribution of the model’s performance over each example, while the radar chart is an intuitive way of comparing different models over various metrics. SummerTime can also output the generated summaries to file formats that are directly compatible with SummVis [50], so that the users can easily use it to visualize the per-instance output differences on the token level.

5 Future Work↩︎

An important piece of future work for SummerTime is to include more summarization models (e.g., multilingual, query-aware, etc) to enlarge the number of choices for the users, and more datasets to increase the chance of users finding similar tasks or domain for evaluation when they do not have a dataset of their own. We also plan to add more visualization methods for the users to better understand the differences between the outputs of various models and the behavior of each individual model itself. Moreover, we would like to enable fine-tuning for a subset of smaller models we support, to enable better performance on some domains or tasks for which no pretrained models are available. With all such potential improvements in the near future, we plan to supply SummerTime not only as a way for non-expert users to access state-of-art summarization models, but also as a go-to choice to quickly establishing baseline results for researchers as well.

6 Conclusion↩︎

We introduce SummerTime, a text summarization toolkit designed for non-expert users. SummerTime includes various summarization datasets, models and evaluation metrics and covers a wide range of summarization tasks. It can also automatically identify the best models or pipelines for a specific dataset and task, and visualize the differences between the model outputs and performances. SummerTime is open source under the Apache-2.0 license and is available online.

Acknowledgements↩︎

The authors would like to thank Rui Zhang, Alexander Fabbri, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz and Rahul Jha for their advice for this work. This work is supported in part by a grant from Microsoft Research.

References↩︎

[1]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 27:3104–3112.

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

[3]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998–6008.

[4]

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.

[5]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.

[6]

Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1–12.

[7]

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. https://doi.org/10.18653/v1/2021.naacl-main.472. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905–5921, Online. Association for Computational Linguistics.

[8]

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. https://doi.org/10.18653/v1/N18-2097. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.

[9]

Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 340–348.

[10]

Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. https://doi.org/10.18653/v1/P19-1102. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics.

[11]

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. https://doi.org/10.18653/v1/D19-5409. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.

[12]

Karl Moritz Hermann, Tomáš Kočiskỳ, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 1693–1701.

[13]

Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541.

[14]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. https://doi.org/10.18653/v1/2020.emnlp-demos.6. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

[15]

Rada Mihalcea and Paul Tarau. 2004. https://www.aclweb.org/anthology/W04-3252. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics.

[16]

Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479.

[17]

Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.

[18]

Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98.

[19]

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, page 280.

[20]

Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1662–1675.

[21]

Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 675–686.

[22]

Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2175–2189.

[23]

Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2019. A closer look at data bias in neural extractive summarization models. EMNLP-IJCNLP 2019, page 80.

[24]

Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Searching for effective neural extractive summarization: What works and what’s next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049–1058.

[25]

Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. Association for Computational Linguistics.

[26]

Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019. Improving the similarity measure of determinantal point processes for extractive multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1027–1038.

[27]

Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. https://doi.org/10.18653/v1/2020.acl-main.552. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6197–6208. Association for Computational Linguistics.

[28]

Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural extractive summarization with hierarchical attentive heterogeneous graph network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3622–3631.

[29]

Kathleen McKeown and Dragomir R Radev. 1995. Generating summaries of multiple news articles. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74–82.

[30]

Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336.

[31]

Hal Daumé III and Daniel Marcu. 2006. https://doi.org/10.3115/1220175.1220214. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 305–312, Sydney, Australia. Association for Computational Linguistics.

[32]

Jahna Otterbacher, Gunes Erkan, and Dragomir R Radev. 2009. Biased lexrank: Passage retrieval using random walks with question-based priors. Information Processing & Management, 45(1):42–54.

[33]

Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardie. 2016. A sentence compression based framework to query-focused multi-document summarization. arXiv preprint arXiv:1606.07548.

[34]

Marina Litvak and Natalia Vanetik. 2017. https://doi.org/10.18653/v1/W17-1004. In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pages 22–31, Valencia, Spain. Association for Computational Linguistics.

[35]

Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. https://doi.org/10.18653/v1/P17-1098. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072, Vancouver, Canada. Association for Computational Linguistics.

[36]

Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704.

[37]

Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694.

[38]

Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021. Summscreen: A dataset for abstractive screenplay summarization. arXiv preprint arXiv:2104.07091.

[39]

Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. Dialsumm: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06762.

[40]

Jiaao Chen and Diyi Yang. 2020. https://www.aclweb.org/anthology/2020.emnlp-main.336. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106–4118, Online. Association for Computational Linguistics.

[41]

Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.19. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 194–203, Online. Association for Computational Linguistics.

[42]

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.

[43]

Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. In International workshop on machine learning for multimodal interaction, pages 28–39. Springer.

[44]

Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–I. IEEE.

[45]

Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7386–7393.

[46]

Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021. http://arxiv.org/abs/2104.06486.

[47]

François Portet, Ehud Reiter, Albert Gatt, Jim Hunter, Somayajulu Sripada, Yvonne Freer, and Cindy Sykes. 2009. Automatic generation of textual summaries from neonatal intensive care data. Artificial Intelligence, 173(7-8):789–816.

[48]

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf. 2021. http://arxiv.org/abs/2109.02846.

[49]

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626.

[50]

Jesse Vig, Wojciech Kryscinski, Karan Goel, and Nazneen Fatema Rajani. 2021. Summvis: Interactive visual analysis of models, data, and evaluation for text summarization. arXiv preprint arXiv:2104.07605.

[51]

Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.

[52]

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.

[53]

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.

[54]

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.647. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational Linguistics.

[55]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

[56]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

[57]

Jun-Ping Ng and Viktoria Abrecht. 2015. https://doi.org/10.18653/v1/D15-1222. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.

[58]

Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pages 228–231.

[59]

Martin F Porter. 2001. Snowball: A language for stemming algorithms.

[60]

George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.

[61]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr. In International Conference on Learning Representations.

[62]

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765–6816.

[63]

Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pages 240–248. PMLR.

https://huggingface.co/models↩︎
https://huggingface.co/datasets↩︎
http://www.summarization.com/mead/↩︎
https://github.com/miso-belica/sumy↩︎
https://github.com/OpenNMT/OpenNMT-py↩︎
Note that in line 9 of the algorithm, the symbol “\(>\)” is conceptual and should be interpreted as “better than”↩︎

SummerTime: Text Summarization Toolkit for Non-experts

Abstract

1 Introduction↩︎

2.1 Text Summarization↩︎

2.2 Existing Systems for Summarization↩︎

3 SummerTime↩︎

3.1 Summarization Models↩︎

Single-document Summarization↩︎

Multi-document Summarization↩︎

Query-based Summarization↩︎

Dialogue Summarization↩︎

3.2 Datasets↩︎

3.3 Evaluation Metrics↩︎

4 Model Selection↩︎

4.0.0.1 Create/select datasets

4.0.0.2 Construct pipelines

4.0.0.3 Search for the best models

4.0.0.4 Visualization

5 Future Work↩︎

6 Conclusion↩︎

Acknowledgements↩︎

References↩︎

Subjects

Updated on Academus

SummerTime: Text Summarization Toolkit for Non-experts

Abstract

1 Introduction↩︎

2 Related Work↩︎

2.1 Text Summarization↩︎

2.2 Existing Systems for Summarization↩︎

3 SummerTime↩︎

3.1 Summarization Models↩︎

Single-document Summarization↩︎

Multi-document Summarization↩︎

Query-based Summarization↩︎

Dialogue Summarization↩︎

3.2 Datasets↩︎

3.3 Evaluation Metrics↩︎

4 Model Selection↩︎

4.0.0.1 Create/select datasets

4.0.0.2 Construct pipelines

4.0.0.3 Search for the best models

4.0.0.4 Visualization

5 Future Work↩︎

6 Conclusion↩︎

Acknowledgements↩︎

References↩︎

Subjects

Updated on Academus