Skip to main content Accessibility help
×
Home

Query-based summarization of discussion threads

  • Suzan Verberne (a1), Emiel Krahmer (a2), Sander Wubben (a2) and Antal van den Bosch (a3) (a4)

Abstract

In this paper, we address query-based summarization of discussion threads. New users can profit from the information shared in the forum, Please check if the inserted city and country names in the affiliations are correct. if they can find back the previously posted information. However, discussion threads on a single topic can easily comprise dozens or hundreds of individual posts. Our aim is to summarize forum threads given real web search queries. We created a data set with search queries from a discussion forum’s search engine log and the discussion threads that were clicked by the user who entered the query. For 120 thread–query combinations, a reference summary was made by five different human raters. We compared two methods for automatic summarization of the threads: a query-independent method based on post features, and Maximum Marginal Relevance (MMR), a method that takes the query into account. We also compared four different word embeddings representations as alternative for standard word vectors in extractive summarization. We find (1) that the agreement between human summarizers does not improve when a query is provided that: (2) the query-independent post features as well as a centroid-based baseline outperform MMR by a large margin; (3) combining the post features with query similarity gives a small improvement over the use of post features alone; and (4) for the word embeddings, a match in domain appears to be more important than corpus size and dimensionality. However, the differences between the models were not reflected by differences in quality of the summaries created with help of these models. We conclude that query-based summarization with web queries is challenging because the queries are short, and a click on a result is not a direct indicator for the relevance of the result.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Query-based summarization of discussion threads
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Query-based summarization of discussion threads
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Query-based summarization of discussion threads
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

*Corresponding author. Email: s.verberne@liacs.leidenuniv.nl

References

Hide All
Aker, A., Paramita, M., Kurtic, E., Funk, A., Barker, E., Hepple, M. and Gaizauskas, R. (2016). Automatic label generation for news comment clusters. In The 9th International Natural Language Generation conference. Edinburgh, UK: Association for Computational Linguistics (ACL), p. 61.
ALMasri, M., Berrut, C. and Chevallet, J.-P. (2016). A comparison of deep learning based query expansion with Pseudo-relevance feedback and mutual information. In Ferro, N., Crestani, F., Moens, M.-F., Mothe, J., Silvestri, F., Di Nunzio, G.M., Hauf, C. and Silvello, G. (eds), Advances in Information Retrieval. Cham: Springer International Publishing, pp. 709-715.
Alonso, O. and Baeza-Yates, R. (2011). Design and implementation of relevance assessments using crowdsourcing. In Clough, P., Foley, C., Gurrin, C., Jones, G.J.F., Kraaij, W., Lee, H. and Mudoch, V. (eds.), Advances in Information Retrieval. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 153-164.
Amini, M.R. and Usunier, N. (2009). Incorporating prior knowledge into a transductive ranking algorithm for multi-document summarization. In Proceedings of the 32Nd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ‘09. New York, NY, USA: ACM, pp. 704-705.
Barker, E., Paramita, M., Aker, A., Kurtic, E., Hepple, M. and Gaizauskas, R. (2016). The SENSEI annotated corpus: Human summaries of reader comment conversations in on-line news. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Los Angeles, CA, USA: Association for Computational Linguistics, pp. 4252.
Bhatia, S., Biyani, P. and Mitra, P. (2014). Summarizing online forum discussions–can dialog acts of individual messages help? In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, pp. 21272131.
Bhatia, S. and Mitra, P. (2010). Adopting inference networks for online thread retrieval. In Fox, M. and Poole, D. (eds), Twenty-Fourth Conference on Artificial Intelligence, Atlanta, Georgia, USA, vol. 10, pp. 13001305.
Cao, Z., Wei, F., Dong, L., Li, S. and Zhou, M. (2015a). Ranking with recursive neural networks and its application to multi-document summarization. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, TX, USA: Association for the Advancement of Artificial Intelligence (AAAI), pp. 21532159.
Cao, Z., Wei, F., Li, S., Li, W., Zhou, M. and Wang, H. (2015b). Learning summary prior representation for extractive summarization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers). Beijing, China: Association for Computational Linguistics, pp. 829833.
Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information retrieval. Melbourne, Australia: ACM, pp. 335336.
Cheng, J. and Lapata, M. (2016). Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany: Association for Computational Linguistics.
Chowdhury, S.A., Calvo, M., Ghosh, A., Stepanov, E.A., Bayer, A.O., Riccardi, G., Garcıa, F. and Sanchis, E. (2015). Selection and aggregation techniques for crowdsourced semantic annotation task. In 16th Annual Conference of the International Speech Communication Association. Dresden, Germany: ISCA, pp. 27792783.
Chowdhury, S.A., Ghosh, A., Stepanov, E.A., Bayer, A.O., Riccardi, G. and Klasinas, I. (2014). Cross-language transfer of semantic annotation via targeted crowdsourcing. In 15th Annual Conference of the International Speech Communication Association. Singapore: ISCA, pp. 21082112.
Condori, R.E.L. and Pardo, T.A.S. (2017). Opinion summarization methods: Comparing and extending extractive and abstractive approaches. Expert Systems with Applications 78, 124134.
Dang, H.T. (2005). Overview of DUC 2005. In Proceedings of the Document Understanding Conference. Vancouver, B.C., Canada: NIST, vol. 2005, pp. 112.
Das, D. and Martins, A.F.T. (2007). A survey on automatic text summarization. Literature Survey for the Language and Statistics II course at CMU 4, 192195.
Denil, M., Demiraj, A., Kalchbrenner, N., Blunsom, P. and de Freitas, N. 2014. Modelling, visualising and summarising documents with a single convolutional neural network. arXiv preprint arXiv:1406.3830.
Dlikman, A. and Last, M. 2016. Using machine learning methods and linguistic features in single-document extractive summarization. In: Proceedings of the Third Edition of the Data Mining and Natural Language Processing (DMNLP) Workshop at ECML/PKDD. Riva del Garda, Italy: INSA Rennes, IRISA, pp. 18.
Dupret, G. and Liao, C. (2010). A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine. Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM). New York City, USA: ACM, pp. 181190.
Erkan, G. and Radev, D.R. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research 22, 457479.
Giannakopoulos, G., Kubina, J., Conroy, J.M., Steinberger, J., Favre, B., Kabadjov, M., Kruschwitz, U. and Poesio, M. (2015). MultiLing 2015: Multilingual summarization of single and multi-documents, on-line fora, and call-center conversations. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). Prague, Czech Republic: Association for Computational Linguistics (ACL), p. 270.
Gong, Y. and Liu, X. (2001). Generic text summarization using relevance measure and latent semantic analysis. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New Orleans, LA, USA: ACM, pp. 1925.
Gupta, V. and Lehal, G.S. (2010). A survey of text summarization extractive techniques. Journal of Emerging Technologies in Web Intelligence 2(3), 258268.
Guy, I. (2016). Searching by talking: Analysis of voice queries on mobile web search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. Pisa, Italy: ACM, pp. 3544.
Hahn, U. and Mani, I. (2000). The challenges of automatic summarization. IEEE Computer 33(11), 2936.
Hovy, E., Lin, C.-Y., Zhou, L. and Fukumoto, J. (2006). Automated summarization evaluation with basic elements. Proceedings of the Fifth Conference on Language Resources and Evaluation (LREC 2006). Genoa, Italy: ELRA, pp. 604611.
Hussain, A., Prakadeswaran, and Prakash, T.S. (2014). Query-based forum posts extraction and refinement. International Journal on Engineering Technology and Sciences – IJETS 1(8), 299304.
Kabadjov, M., Steinberger, J., Barker, E., Kruschwitz, U. and Poesio, M. (2015). OnForumS: The shared task on online forum summarisation at MultiLing’15. Proceedings of the 7th Forum for Information Retrieval Evaluation (FIRE). Gandhinagar, India: ACM, pp. 2126.
Kekäläinen, J. and Järvelin, K. (2002). Using graded relevance assessments in IR evaluation. Journal of the Association for Information Science and Technology 53(13), 11201129.
Kenter, T., Borisov, A. and de Rijke, M. (2016). Siamese CBOW: Optimizing word embeddings for sentence representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1. Berlin, Germany: Association for Computational Linguistics, pp. 941951.
Krishnamani, J., Zhao, Y. and Sunderraman, R. (2013). Forum summarization using topic models and content-metadata sensitive clustering. In Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)-Volume 03. Washington, DC, USA: IEEE Computer Society, pp. 195198.
Kupiec, J., Pedersen, J. and Chen, F. (1995). A trainable document summarizer. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, WA, USA: ACM, pp. 6873.
Kusner, M., Sun, Y., Kolkin, N. and Weinberger, K. (2015). From word embeddings to document distances. In International Conference on Machine Learning (ICML), vol. 15. Lille, France. pp. 957966. http://proceedings.mlr.press.
Landis, J. and Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159174.
Li, Y. and Li, S. (2014). Query-focused multi-document summarization: Combining a topic model with graph-based semi-supervised learning. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin, Ireland: Association for Computational Linguistics, pp. 11971207.
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Moens, M.-F. and Szpakowicz, S. (eds.), Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Barcelona, Spain: Association for Computational Linguistics, pp. 7481.
Lin, C.-Y. and Hovy, E. (2000). The automated acquisition of topic signatures for text summarization. Proceedings of the 18th Conference on Computational Linguistics-Volume 1. Saarbrücken, Germany: Association for Computational Linguistics, pp. 495501.
Liu, F. and Liu, Y. 2008. Correlation between rouge and human evaluation of extractive meeting summaries. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Columbus, Ohio: Association for Computational Linguistics, pp. 201204.
Llewellyn, C., Grover, C. and Oberlander, J. (2014). Summarizing newspaper comments. Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media. Palo Alto, California: Association for the Advancement of Artificial Intelligence (AAAI), pp. 599602.
Marge, M., Banerjee, S. and Rudnicky, A.I. (2010). Using the Amazon Mechanical Turk to transcribe and annotate meeting speech for extractive summarization. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. Los Angeles, California: Association for Computational Linguistics, pp. 99107.
Mehdad, Y., Carenini, G. and Ng, R.T. (2014). Abstractive summarization of spoken and written conversations based on phrasal queries. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 12201230.
Metzler, D. and Kanungo, T. (2008). Machine learned sentence selection strategies for query-biased summarization. In Li, H., Liu, T.-Y. and Zhai, C. X. (eds.), Proceedings of the SIGIR 2008 Workshop “Learning to Rank for Information Retrieval”, Singapore: Microsoft Research, pp. 4047.
Mikolov, T., Chen, K., Corrado, G. and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Mohamed, A.A. and Rajasekaran, S. (2006). Improving query-based summarization using document graphs. In 2006 IEEE International Symposium on Signal Processing and Information Technology. Vancouver, Canada: IEEE, pp. 408410.
Murray, G., Renals, S. and Carletta, J. (2005). Extractive summarization of meeting recordings. In INTERSPEECH-2005. ISCA, Edinburgh, UK, pp. 593596.
Nallapati, R., Zhai, F. and Zhou, B. (2016). SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents. arXiv preprint arXiv:1611.04230.
Nallapati, R., Zhou, B. and Ma, M. (2016). Classify Or select: neural architectures for extractive document summarization. arXiv preprint arXiv:1611.04244v1.
Nenkova, A. and McKeown, K. (2011). Automatic summarization. Foundations and Trends in Information Retrieval 5 (23), 103233.
Nenkova, A. and McKeown, K. (2012). A survey of text summarization techniques. In Aggarwal, C. C. and Zhai, C. X. (eds.), Mining Text Data. Springer, Switzerland: Springer Nature, pp. 4376.
Oya, T., Mehdad, Y., Carenini, G. and Ng, R. (2014). A template-based abstractive meeting summarization: Leveraging summary and source text relationships. Proceedings of the 8th International Natural Language Generation Conference (INLG), Philadelphia, PA, USA: The Association for Computational Linguistics, pp. 4553.
Park, S., Lee, J.-H., Ahn, C.-M., Hong, J.S. and Chun, S.-J. (2006). Query based summarization using non-negative matrix factorization. In International Conference on Knowledge-Based and Intelligent Information and Engineering Systems. Bournemouth, United Kingdom: Springer, pp. 8489.
Parthasarathy, S. and Hasan, T. (2015). Automatic broadcast news summarization via rank classifiers and crowdsourced annotation. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 52565260.
Pembe, F.C. and Güngör, T. (2007). Automated query-biased and structure-preserving text summarization on web documents. In Proceedings of the International Symposium on Innovations in Intelligent Systems and Applications, I˙stanbul.
Penn, G. and Zhu, X. (2008). A critical reassessment of evaluation baselines for speech summarization. In Proceedings of ACL-08: HLT. Columbus, OH, USA: Association for Computational Linguistics, pp. 470478.
Powers, D.M.W. (2012). The problem with kappa. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Avignon, France: Association for Computational Linguistics, pp. 345355.
Radev, D.R., Jing, H., Sty´s, M. and Tam, D. (2004). Centroid-based summarization of multiple documents. Information Processing and Management 40 (December), 919938.
Radev, D.R., Teufel, S., Saggion, H., Lam, W., Blitzer, J., Qi, H., Celebi, A., Liu, D. and Drabek, E. (2003). Evaluation challenges in large-scale document summarization. Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1. Sapporo, Japan: Association for Computational Linguistics, pp. 375382.
Ren, Z., Ma, J., Wang, S. and Liu, Y. (2011). Summarizing web forum threads based on a latent topic propagation process. Proceedings of the 20th ACM International Conference on Information and Knowledge Management. Glasgow, United Kingdom: ACM, pp. 879884.
Schilder, F. and Kondadadi, R. (2008). FastSum: fast and accurate query-based multi-document summarization. In Proceedings of the 46th annual meeting of the association for computational linguistics on human language technologies: Short papers. Association for Computational Linguistics, pp. 205208.
Shen, C. and Li, T. (2011). Learning to rank for query-focused multi-document summarization. In 2011 IEEE 11th International Conference on Data Mining (ICDM). IEEE, pp. 626634.
Sipos, R., Shivaswamy, P. and Joachims, T. (2012). Large-margin learning of submodular summarization models. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 224233.
Svore, K.M., Vanderwende, L. and Burges, C.J.C. (2007). Enhancing Single-Document Summarization by Combining RankNet and Third-Party Sources. EMNLP-CoNLL, pp. 448457.
Teevan, J., Ramage, D. and Morris, M.R. (2011). # TwitterSearch: a comparison of microblog search and web search. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining. Kowloon, Hong Kong: ACM, pp. 3544.
Tigelaar, A.S., op den Akker, R. and Hiemstra, D. (2010). Automatic summarisation of discussion fora. Natural Language Engineering 16(2), 161192.
Tombros, A. and Sanderson, M. (1998). Advantages of query biased summaries in information retrieval. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Melbourne, Australia: ACM, pp. 210.
Toutanova, K., Brockett, C., Gamon, M., Jagarlamudi, J., Suzuki, H. and Vanderwende, L. (2007). The pythy summarization system: Microsoft research at DUC 2007. In Proceedings of the Document Understanding Conference (DUC). Rochester, New York, USA: NIST.
Tsai, C.-I., Hung, H.-T., Chen, K.-Y. and Chen, B. (2016). Extractive speech summarization leveraging convolutional neural network techniques. In: Proceedings of 2016 IEEE Workshop on Spoken Language Technology. IEEE: San Diego, California
Tulkens, S., Emmery, C. and Daelemans, W. (2016). Evaluating unsupervised Dutch word embeddings as a linguistic resource. In Calzolari, N., Choukri, K., Declerck, T., Grobelnik, M., Maegaard, B., Mariani, J., Moreno, A., Odijk, J. and Piperidis, S. (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). Paris, France: European Language Resources Association (ELRA).
van Oortmerssen, G., Raaijmakers, S., Sappelli, M., Boertjes, E., Verberne, S., Walasek, N. and Kraaij, W. (2017). Analyzing cancer forum discussions with text mining. Knowledge Representation for Health Care Process-Oriented Information Systems in Health Care Extraction & Processing of Rich Semantics from Medical Texts, p. 127.
van Uden-Kraan, C.F., Drossaert, C.H.C., Taal, E., Seydel, E.R. and van de Laar, M.A.F.J. (2008). Self-reported differences in empowerment between lurkers and posters in online patient support groups. Journal of Medical Internet Research 10(2). https://protect-eu.mimecast.com/s/zyL4CEqpqtWQNpQCQ9wtT?domain=dx.doi.org”10.2196/jmir.992
van Uden-Kraan, C.F., Drossaert, C.H.C., Taal, E., Seydel, E.R. and van de Laar, M.A.F.J. (2009). Participation in online patient support groups endorses patients empowerment. Patient Education and Counseling 74(1), 6169.
Verberne, S., Heijden, M., Hinne, M., Sappelli, M., Koldijk, S., Hoenkamp, E. and Kraaij, W. (2013). Reliability and validity of query intent assessments. Journal of the American Society for Information Science and Technology 64(11), 22242237.
Verberne, S., Krahmer, E., Hendricks, I., Wubben, S. and Van den Bosch, A. (2017). Creating a reference data set for the summarization of discussion forum threads. Language Resources and Evaluation, 52(2), 461483.
Wan, X. and Peng, Y. (2005). The earth mover’s distance as a semantic measure for document similarity. Proceedings of the 14th ACM International Conference on Information and Knowledge Management. Bremen, Germany: ACM, pp. 301302.
Weimer, M., Gurevych, I. and Mühlhäuser, M. (2007). Automatically assessing the post quality in online discussions on software. Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions. Prague, Czech Republic: Association for Computational Linguistics, pp. 125128.
Xu, S. and Lorber, M.F. (2014). Interrater agreement statistics with skewed data: Evaluation of alternatives to Cohen’s kappa. Journal of Consulting and Clinical Psychology 82(6), 1219.
Yin, W. and Pei, Y. (2015). Optimizing sentence modeling and selection for document summarization. Proceedings of the 24th International Conference on Artificial Intelligence. Buenos Aires, Argentina: Association for the Advancement of Artificial Intelligence (AAAI), pp. 13831389.
Zajic, D.M., Dorr, B.J. and Lin, J. (2008). Single-document and multi-document summarization techniques for email threads using sentence compression. Information Processing and Management 44(4), 16001610.
Zhang, R., Li, W., Gao, D. and Ouyang, Y. (2013). Automatic twitter topic summarization with speech acts. IEEE Transactions on Audio, Speech, and Language Processing 21(3), 649658.
Zhang, Y., Er, M.J., Zhao, R. and Pratama, M. (2016). Multiview convolutional neural networks for multidocument extractive summarization. IEEE Transactions on Cybernetics, 47(10), 32303242.
Zhou, L. and Hovy, E. (2005). Digesting virtual geek culture: The summarization of technical internet relay chats. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Ann Arbor, Michigan: Association for Computational Linguistics, pp. 298305.
Zhou, L. and Hovy, E.H. (2006). On the summarization of dynamically introduced information: Online discussions and blogs. AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs. Association for the Advancement of Artificial Intelligence (AAAI), pp. 237246.

Keywords

Query-based summarization of discussion threads

  • Suzan Verberne (a1), Emiel Krahmer (a2), Sander Wubben (a2) and Antal van den Bosch (a3) (a4)

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed