Hostname: page-component-76c49bb84f-lxspr Total loading time: 0 Render date: 2025-07-04T21:09:06.625Z Has data issue: false hasContentIssue false
  • English
  • Français

Designing a virtual patient dialogue system based on terminology-rich resources: Challenges and evaluation

Published online by Cambridge University Press:  15 July 2019

Leonardo Campillos-Llanos Open the ORCID record for Leonardo Campillos-Llanos [Opens in a new window] ,
Catherine Thomas ,
Éric Bilinski ,
Pierre Zweigenbaum  and
Sophie Rosset
Leonardo Campillos-Llanos*
Affiliation:
LIMSI, CNRS, Université Paris-Saclay, Orsay, France SATT Paris-Saclay, Orsay, France
Catherine Thomas
Affiliation:
LIMSI, CNRS, Université Paris-Saclay, Orsay, France SATT Paris-Saclay, Orsay, France
Éric Bilinski
Affiliation:
LIMSI, CNRS, Université Paris-Saclay, Orsay, France
Pierre Zweigenbaum
Affiliation:
LIMSI, CNRS, Université Paris-Saclay, Orsay, France
Sophie Rosset
Affiliation:
LIMSI, CNRS, Université Paris-Saclay, Orsay, France
*
*Corresponding author. E-mails: campillos@limsi.fr; leonardo.campillos@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Virtual patient software allows health professionals to practise their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history-taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non-medical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8%. Dialogue management provided on average 74.3 (±9.5)% of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8% of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.

Keywords

Dialogue systemVirtual patientTerminologyLanguage resources

Information

Type
Article
Information
Natural Language Engineering , Volume 26 , Issue 2 , March 2020 , pp. 183 - 220
Copyright
© Cambridge University Press 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Bates, B. and Bickley, L.S. 2014. Guide de l’examen clinique – Nouvelle édition 2014. London/Montrouge: Arnette-John Libbey Eurotext.Google Scholar
Beveridge, M. and Fox, J. 2006. Automatic generation of spoken dialogue from medical plans and ontologies. Journal of Biomedical Informatics 39(5), 482–499.CrossRefGoogle ScholarPubMed
Benedict, N. 2010. Virtual patients and problem-based learning in advanced therapeutics. American Journal of Pharmaceutical Education 74(8), article 143.CrossRefGoogle ScholarPubMed
Bickmore, T. 2015. Conversational agents for automated inpatient and outpatient health counseling. In Proc. of the AMIA Symposium, San Francisco, USA, p. 2131.Google Scholar
Bickmore, T. and Giorgino, T. 2006. Health dialog systems for patients and consumers. Journal of Biomedical Informatics 39(5), 556–571.CrossRefGoogle ScholarPubMed
Bodenreider, O. 2004. The Unified Medical Language System (UMLS): Integrating biomedical terminology. Nucleic Acids Research 32(suppl 1), D267–D270.CrossRefGoogle ScholarPubMed
Bouamor, D., Campillos-Llanos, L., Ligozat, A.-L., Rosset, S. and Zweigenbaum, P. 2016. Transfer-based learning-to-rank assessment of medical term technicality. In Calzolari, N.et al. (eds.), Proc. of LREC 2016, Portorož, Slovenia, pp. 2312–2316.Google Scholar
Campillos-Llanos, L., Bouamor, D., Bilinski, E., Ligozat, A.-L., Zweigenbaum, P. and Rosset, S. 2015. Description of the PatientGenesys dialogue system. In Proc. of SIGDIAL, Prague, Czech Republic, pp. 438–440.Google Scholar
Campillos-Llanos, L., Bouamor, D., Zweigenbaum, P. and Rosset, S. 2016. Managing linguistic and terminological variation in a medical dialogue system. In Calzolari, N.et al. (eds.), Proc. of LREC 2016, Portorož, Slovenia, pp. 3167–3173.Google Scholar
Campillos-Llanos, L., Rosset, S. and Zweigenbaum, P. 2017. Automatic classification of doctor-patient questions for a virtual patient record query task. In Proc. of the 16th BioNLP 2017 Workshop, Vancouver, Canada, pp. 333–341.CrossRefGoogle Scholar
Celikyilmaz, A., Deng, L. and Hakkani-Tur, D. 2017. Deep Learning for Spoken and Text Dialog Systems. In Deng, L. and Liu, Y. (eds.) Deep Learning in Natural Language Processing, Berlin: Springer, pp. 49–78.Google Scholar
Cole, R. 1999. Tools for research and education in speech science. In Proc. of the International Conference of Phonetic Sciences, San Francisco, USA, vol. 1, pp. 277–281.Google Scholar
Cook, D.A., Erwin, P.J. and Triola, M.M. 2010. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Academic Medicine bf 85(10): 1589–1602.CrossRefGoogle ScholarPubMed
Coudé, C., Coudé, F.-X., and Kassmann, K. 2011. Guide de conversation médicale - français-anglais-allemand. Paris: Lavoisier.Google Scholar
Courtois, B. 1990. Un système de dictionnaires électroniques pour les mots simples du français. Langue française 87(1), 11–22.CrossRefGoogle Scholar
Danforth, D.R., Procter, M., Chen, R., Johnson, M. and Heller, R. 2009. Development of virtual patient simulations for medical education. Journal For Virtual Worlds Research 2(2), 4–11.CrossRefGoogle Scholar
Datta, D., Brashers, V., Owen, J., White, C. and Barnes, L. 2016. A Deep Learning Methodology for Semantic Utterance Classification in Virtual Human Dialogue Systems. In Proc. of the International Conference on Intelligent Virtual Agents 2016, Berlin: Springer-Verlag, pp. 451–455.Google Scholar
Dickerson, R., Johnsen, K., Raij, A., Lok, B., Hernandez, J., Stevens, A. and Lind, D.S. 2005. Evaluating a script-based approach for simulating patient-doctor interaction. In Proc. of the Intern. Conference of Human-Computer Interface Advances for Modeling and Simulation, pp. 79–84.Google Scholar
Donnelly, K. 2006. SNOMED-CT: The advanced terminology and coding system for eHealth. Studies in Health Technology and Informatics 121, 279–290.Google ScholarPubMed
Ellaway, R., Candler, C., Greene, P. and Smothers, V. 2006. An architectural model for MedBiquitous virtual patients. http://groups.medbiq.org/medbiq/display/VPWG/MedBiquitous+Virtual+Patient+Architecture. Accessed 23 April 2018.Google Scholar
Epstein, O., Perkin, D., Cookson, J., and de Bono, D.P. 2015. Guide pratique de l’examen clinique. Paris: Elsevier Masson.Google Scholar
Galibert, O. 2009. Approaches and methodologies for automatic Question-Answering in an open-domain, interactive setup. Phd dissertation, Université Paris Sud - Paris XI.Google Scholar
Giorgino, T., Azzini, I., Rognoni, C., Quaglini, S., Stefanelli, M., Gretter, R., and Falavigna, D. 2005. Automated spoken dialogue system for hypertensive patient home management. International Journal of Medical Informatics 74(2), 159–167.CrossRefGoogle ScholarPubMed
Gokcen, A., Jaffe, E., Erdmann, J., White, M. and Danforth, D. 2016. A corpus of word-aligned asked and anticipated questions in a virtual patient dialogue system. In Calzolari, N.et al. (eds.), Proc. of LREC 2016, Portorož, Slovenia, pp. 3174–3179.Google Scholar
Hathout, N., Namer, F. and Dal, G. 2002. An experimental constructional database: the MorTAL project. In Hathout, N., Namer, F. and Dal, G. (eds.) Many Morphologies, Somerville, MA: Cascadilla Press, pp. 178–209.Google Scholar
Hoxha, J., and Weng, C. 2016. Leveraging dialog systems research to assist biomedical researchers’ interrogation of Big Clinical Data. Journal of Biomedical Informatics 61, 176–184.CrossRefGoogle ScholarPubMed
Hubal, R.C., Kizakevich, P.N., Guinn, C.I., Merino, K.D. and West, S.L. 2000. The virtual standardized patient. Studies in Health Technology and Informatics 70, 133–138.Google ScholarPubMed
Hubal, R.C., Deterding, R.R., Frank, G.A., Schwetzke, H.F. and Kizakevich, P.N. 2003. Lessons learned in modeling virtual pediatric patients. Studies in Health Technology and Informatics 94, 127–130.Google ScholarPubMed
Jin, L., White, M., Jaffe, E., Zimmerman, L. and Danforth, D. 2017. Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System. In Proc. of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, Copenhagen, Denmark, pp. 11–21.Google Scholar
Jokinen, K. and McTear, M. 2009. Spoken Dialogue Systems. Synthesis Lectures on Human Language Technologies, 2. San Rafael, CA: Morgan and Claypool Publishers.CrossRefGoogle Scholar
Kenny, P., Parsons, T.D., Gratch, J. and Rizzo, A.A. 2008. Evaluation of Justina: a virtual patient with PTSD. In Prendinger, H., Lester, J., and Ishizuka, M. (eds.), Proc. of Intelligent Virtual Agents, Berlin: Springer-Verlag, pp. 394–408.CrossRefGoogle Scholar
Kenny, P. and Parsons, T. 2011. Embodied conversational virtual patients. In Perez-Marín, D. and Pascual Nieto, I. (eds.) Conversational Agents and Natural Language Interaction: Techniques and Effective Practices, Hershey: IGI Global, pp. 254–281.CrossRefGoogle Scholar
Lelardeux, C., Panzoli, D., Alvarez, J., Galaup, M. and Lagarrigue, P. 2013. Serious game, simulateur, serious play:état de l’art pour la formation en santé. In Actes du colloque Serious Games en Médecine et Santé (SeGaMED) 2013, Nice: e-virtuoses, pp. L3/27–38.Google Scholar
Levenshtein, V.I. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10(8), 707–710.Google Scholar
Maicher, K., Danforth, D., Price, A., Zimmerman, L., Wilcox, B., Liston, B, Cronau, H., Belknap, L., Ledford, C., Way, D., Post, D., Macerollo, A. and Rizer, M. 2017. Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills. Simulation in Healthcare 12(2), 124–131.CrossRefGoogle ScholarPubMed
Makhoul, J., Kubala, F., Schwartz, R. and Weischedel, R. 1999. Performance measures for information extraction. In Proc. of DARPA Broadcast News Workshop, Virginia, USA, pp. 249–252.Google Scholar
McCray, A.T., Srinivasan, S. and Browne, A.C. 1994. Lexical methods for managing variation in biomedical terminologies. In Proc. of Annual Symposium Computer Applic. Medical Care, Washington, pp. 235–239.Google Scholar
McCray, A.T., Burgun, A. and Bodenreider, O. 2001. Aggregating UMLS semantic types for reducing conceptual complexity. Studies in Health Technology and Informatics 84, 216–220.Google ScholarPubMed
McTear, M., O’Neill, I., Hanna, P. and Liu, X. 2005. Handling errors and determining confirmation strategies–an object-based approach. Speech Communication 45(3), 249–269.CrossRefGoogle Scholar
Nadkarni, P., Chen, R. and Brandt, C. 2001. UMLS concept indexing for production databases: a feasibility study. Journal of the American Medical Informatics Association 8(1), 80–91.CrossRefGoogle ScholarPubMed
Namer, F. and Zweigenbaum, P. 2004. Acquiring meaning for French medical terminology: contribution of morphosemantics. In Proc. of the 11th MEDINFO Conference, San Francisco, USA, pp. 535–539.Google Scholar
Nirenburg, S., Beale, S., McShane, M., Jarrell, B. and Fantry, G. 2008. Language understanding in Maryland virtual patient. In Proc. of the 22nd International Conference on Computational Linguistics, pp. 36–39.Google Scholar
Nirenburg, S., McShane, M., Beale, S. and Jarrell, B. 2008. Adaptivity in a multi-agent clinical simulation system. In Proc. of AKRR’08, International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pp. 17–19.Google Scholar
Nirenburg, S., McShane, M. and Beale, S. 2009. A unified ontological-semantic substrate for physiological simulation and cognitive modeling. In Proc. of International Conference on Biomedical Ontology (ICBO), Buffalo, New York, pp. 139–142.Google Scholar
Norvig, P. 2007. How to write a spelling corrector. http://norvig.com/spell-correct.html. Accessed 23 April 2018.Google Scholar
Paek, T. 2001. Empirical methods for evaluating dialog systems. In Proc. of the workshop on Evaluation for Language and Dialogue Systems-Volume 9, Toulouse, France, pp. 1–9.Google Scholar
Pastore, F. 2015. How can I help you today? Guide de la consultation médicale et paramédicale en anglais. Paris: Ellipses.Google Scholar
Patrick, J. and Li, M. 2012. An ontology for clinical questions about the contents of patient notes. Journal of Biomedical Informatics 45(2), 292–306.CrossRefGoogle ScholarPubMed
Pinault, F. 2011. Apprentissage par renforcement pour la généralisation des approches automatiques dans la conception des systemes de dialogue oral. PhD dissertation, Avignon University, Avignon, France.Google Scholar
Purver, M., Ginzburg, J. and Healey, P. 2003. On the means for clarification in dialogue. In van Kuppevelt, J. and Smith, R. W. (eds.) Current and new directions in discourse and dialogue, Dordrecht: Springer, pp. 235–255.CrossRefGoogle Scholar
Quirk, R., Crystal, D., Greenbaum, S., Leech, G. and Svartvik, J. (1985). A comprehensive grammar of the English language. New York: Longman.Google Scholar
Rombauts, N. 2014. Patients virtuels: pédagogie, état de l’art et développement du simulateur Alphadiag. PhD dissertation, Faculty of Medicine, Claude Bernard University, Lyon, France.Google Scholar
Rossen, B., Lind, S. and Lok, B. 2009. Human-centered distributed conversational modeling: Efficient modeling of robust virtual human conversations. In Ruttkay, Z.et al. (eds.) Proc. of the International Workshop on Intelligent Virtual Agents, Berlin: Springer, pp. 474–481.CrossRefGoogle Scholar
Rossen, B. and Lok, B. 2012. A crowdsourcing method to develop virtual human conversational agents. International Journal of Human-Computer Studies 70(4), 301–319.CrossRefGoogle Scholar
Rosset, S., Galibert, O., Illouz, G. and Max, A. Integrating Spoken Dialog and Question Answering: the Ritel Project Proc. of InterSpeech 2006, Pittsburgh, USA, pp. 1914–1917.Google Scholar
Rosset, S., Galibert, O., Adda, G. and Bilinski, E. 2008. The LIMSI participation in the QAst track. In Advances in Multilingual and Multimodal Information Retrieval, Berlin: Springer-Verlag, pp. 414–423.CrossRefGoogle Scholar
Roy, B. and Graham, T.N. 2008. Methods for evaluating software architecture: A survey. Technical Report 545, School of Computing, Queen’s University at Kingston, Ontario, Canada.Google Scholar
Salazar, V.L., Eisman Cabeza, E. M., Castro Peña, J. L. and Zurita, J.M. 2012. A case based reasoning model for multilingual language generation in dialogues. Expert Systems with Applications 39(8), 7330–7337.CrossRefGoogle Scholar
Siregard, P., Julen, N. and Lessard, Y. 2013. Apprendre le raisonnement clinique par jeu sérieux. In Actes du colloque Serious Games en Médecine et Santé (SeGaMED) 2013, Nice: e-virtuoses, pp. 79–83.Google Scholar
Stevens, A., Hernandez, J., Johnsen, K., Dickerson, R., Raij, A., Harrison, C., DiPietro, M., Allen, B., Ferdig, R., Foti, S., et al. 2006. The use of virtual patients to teach medical students history taking and communication skills. The American Journal of Surgery 191(6), 806–811.CrossRefGoogle ScholarPubMed
Sutton, R.S., and Barto, A.G. 1998. Reinforcement Learning: An Introduction. Cambridge: MIT press.Google Scholar
Talbot, T.B., Sagae, K., John, B. and Rizzo, A.A. 2012a. Sorting out the virtual patient: how to exploit artificial intelligence, game technology and sound educational practices to create engaging role-playing simulations. International Journal of Gaming and Computer-Mediated Simulations 4(3), 1–19.CrossRefGoogle Scholar
Talbot, T.B., Sagae, K., John, B., Rizzo, A.A. and Playa, C. 2012b. Designing useful virtual standardized patient encounters. In Proc. of the Interservice/Industry Training, Simulation and Education Conference 4(3), 3–6.Google Scholar
Talbot, T.B., Kalisch, N., Christoffersen, K., Lucas, G. and Forbell, E. 2016. Natural language understanding performance and use considerations in virtual medical encounters. Studies in Health Technology and Informatics 220, 407–413.Google ScholarPubMed
Traum, D.R. and Larsson, S. 2003. The information state approach to dialogue management. In van Kuppevelt, J. and Smith, R. W. (eds.) Current and new directions in discourse and dialogue, Dordrecht: Springer, pp. 325–353.CrossRefGoogle Scholar
Traum, D.R., Robinson, S. and Stefan, J. 2004. Evaluation of a multi-party virtual reality dialogue interaction. In Proc. of LREC 2004, Lisbon, Portugal, pp. 1699–1702.Google Scholar
van Schooten, B., Rosset, S., Galibert, O., Max, A., op den Akker, R. and Illouz, G. 2007. Handling speech input in the Ritel QA dialogue system. In Proc. of Interspeech, Antwerp, Belgium, pp. 126–129.Google Scholar
Walker, M.A., Litman, D.J., Kamm, C.A. and Abella, A. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proc. of the 8th Conference of the European chapter of the Association for Computational Linguistics, Madrid, Spain, pp. 271–280.Google Scholar
Young, S.J. 2006. Using POMDPs for dialog management. In Proc. of Spoken Language Technology Workshop, Palm Beach, Aruba, pp. 8–13.Google Scholar
Zweigenbaum, P., Baud, R.H., Burgun, A., Namer, F., Jarrousse, É., Grabar, N., Ruch, P., Le Duff, F., Forget, J.-F., Douyère, M. and Darmoni, S. 2005. A unified medical lexicon for French. International Journal of Medical Informatics 74(2–4), 119–124.CrossRefGoogle ScholarPubMed