Access to healthcare profoundly impacts the health and quality of life of Deaf people. Automatic translation tools are crucial in improving communication between Deaf patients and their healthcare providers. The aim of this chapter is to present the pipeline used to create the Swiss-French Sign Language (LSF-CH) version of BabelDr, a speech-enabled fixed-phrase translator that was initially conceived to improve communication in emergency settings between doctors and allophone patients (Bouillon et al, 2021). In order to do so, we start off by explaining how we ported BabelDr in LSF-CH using both human and avatar videos. We first describe the creation of a reference corpus consisting of video translations done by human translators, then we present a second corpus of videos generated with a virtual human. Finally, we relate the findings of a questionnaire on Deaf users’ perspective on the use of signing avatars in the medical context. We showed that, although respondents prefer human videos, the use of automatic technologies associated with virtual characters is not without interest to the target audience and can be useful to them in the medical context.
6.1 Introduction
According to the Swiss Federal Statistical Office, 9 percent of the population speaks a language not among the four national languages. Moreover, one-third of this 9 percent understands none of the national languages. If these people are ill and require treatment, language barriers can pose considerable obstacles to their care, from both clinical and ethical viewpoints. Clearly, this issue hugely impacts equal access to healthcare (Reference Flores, Laws and MayoFlores et al., 2003; Reference Wasserman, Renfrew and GreenWasserman et al., 2014). One way to provide quality healthcare to all and to facilitate communication between doctors and patients is through the use of translation technologies – more specifically, by using fixed-phrase translators, now widely used in the medical field (see Chapter 5). Although ideal tools would provide the flexibility of full machine translation systems, various studies show that the fixed-phrase translation systems currently available can offer good alternatives to full machine translation for such safety-critical domains (Reference Bouillon, Gerlach, Spechbach, Tsourakis and Halimi.Bouillon et al., 2017; Reference Turner, Choi and DewTurner et al., 2019).
BabelDr is a flexible speech-enabled phraselator aimed at language barrier-related problems in emergency settings (Reference Bouillon, David, Strasly and Spechbach.Bouillon et al., 2021). BableDr is now in use for immigrants speaking non-national languages; however, the application is also under development at present for the local Deaf linguistic minority. We refer here to Deaf patients who live in the French-speaking area of Switzerland and use Swiss-French Sign Language (LSF-CH) as their mother tongue or preferred language. Deaf LSF-CH users identify as members of a minority community with its own language and culture (Reference Padden and Humphries.Padden and Humphries, 1988; Reference PrestonPreston, 1995). The use of the capital D in “Deaf” refers to their cultural identity.
Research in past years has shown that access to healthcare impacts the health and quality of life of Deaf people. Although the need for enhanced access to healthcare services has been highlighted (Reference Emond, Ridd and SutherlandEmond et al., 2015; Reference Kuenburg, Fellinger and Fellinger.Kuenburg et al., 2016), the issue remains quite challenging, even in high-income countries (Reference Pollard, Betts and CarrollPollard et al., 2014; Reference Smeijers and Pfau.Smeijers and Pfau, 2009). Much like ethnic minority groups, Deaf people encounter severe barriers when trying to communicate in a healthcare context. The associated miscommunication between patients and their healthcare providers can lead to potential misunderstandings of diagnosis and treatment (Reference ScheierScheier, 2009) and to a lack of trust. In England, a report by the Royal National Institute for Deaf People recounts the experiences of various Deaf people using health services. Sixty-six percent of British Sign Language users find communication with staff difficult; thirty percent avoid visiting their family doctor for communication reasons; and 33 percent remain unsure about instructions or about the correct treatment following consultations with family doctors (Reference Abou-Abdallah and Lamyman.Abou-Abdallah and Lamyman, 2021; Reference Middleton, Niruban, Girling and Myint.Middleton et al., 2010). Similar results have been shown in the Netherlands, where a study found that 39 percent of Deaf patients who took part in the survey rated their communication with healthcare practitioners as moderate or bad (Reference Smeijers and Pfau.Smeijers and Pfau, 2009), and in the USA where Deaf patients report great difficulties in communication with their physicians (Reference Ralston, Zazove and GorenfloRalston, Zazove and Gorenflo, 1996). In Switzerland, studies carried out by Tatjana Binggeli in 2015 and by Odile Cantero in 2016 also highlight similar barriers (Reference BinggeliBinggeli, 2015; Reference CanteroCantero, 2016). While additional projects have addressed the healthcare needs of Deaf people in Switzerland, communication barriers still remain today (Strasly, in preparation).
Improvements are certainly achievable. For instance, provision of specific training in cultural competency by knowledgeable community representatives could make healthcare professionals more aware of their communication preferences. In recent years, rapid advances in the use and performance of information technology have also greatly benefited Deaf people, and have the potential to make healthcare more accessible to this community and thus enable them to receive adequate and equal care. The aim of this chapter is to focus on the development of speech-enabled fixed-phrase translators for sign languages and, more specifically, to present the pipeline used to create the LSF-CH version of the BabelDr system. While phraselators such as MediBabble (medibabble.com) and Universal Doctor (universaldoctor.com) are commonly used in medicine (Reference Khander, Farag and Chen.Khander et al., 2018; Reference Panayiotou, Gardner and WilliamsPanayiotou et al., 2019), they rarely integrate sign language. Development of such a pipeline is therefore a necessary step toward collection of corpora and creation of useful translation tools.
In the following sections, we first give an outline of the current legal framework regarding the right to health and access to healthcare in Switzerland in order to elucidate the legal background favorable to our project’s emergence. A brief overview of the core principles of the “right to health” follows. We then describe existing sign language projects aimed at improving doctor-patient communication. While some translation tools do exist, they are always limited to very specific coverage, are often unsophisticated, and provide no general solutions for production of sign language resources and translation into sign language. We then explain how we ported BabelDr for LSF-CH using both human and avatar videos. Finally, we present the results of a questionnaire about Deaf users’ perspective on the use of signing avatars in the medical context.
6.2 Legal Framework in Switzerland
Currently, there are no precise or official statistics concerning the number of profoundly Deaf individuals in Switzerland. Current estimates are based on the following formula, established by the World Health Organization (WHO) and used worldwide: number of Deaf signing people = 0.001 percent of the total population, i.e., 1 per 1000 inhabitants. Based upon this formula, and upon the numbers of (1) memberships of Deaf people in clubs and associations and (2) users of interpreting services, Deaf sign language users in all of Switzerland currently number approximately 10,000 people (Reference Braem, Rathmann. and BrentariBraem and Rathmann, 2010). Three different sign languages are used: Swiss-German Sign Language (DSGS) is used in the German-speaking area of Switzerland; LSF-CH in western Switzerland; and Swiss-Italian Sign Language (LIS-SI) in the Italian-speaking region.
Each region formerly had its own association of Deaf people. These were then federated in 2006 under an umbrella organization, the Swiss Federation of the Deaf, which strives to achieve equal rights for the Deaf and hard-of-hearing throughout the country. Per its new strategic plan for 2021–25, the Federation will undertake four key areas of action, as voted by members in October 2020: (1) inclusion in the labor market; (2) participation in direct democracy; (3) access to the healthcare system; and (4) inclusive education (SGB-FSS, 2021). Concerning access to the healthcare system, discrimination against Deaf people is not due to a lack of legislation (Reference Binggeli, Hohenstein., Hohenstein and Lévy-TödterBinggeli and Hohenstein, 2020). In fact, Switzerland has signed international treaties and has enacted national and cantonal legislation that promotes the highest health standards for its population. Instead, the challenges probably stem from the country’s federal makeup (Reference Marks-Sultan, Kurt, Leyvraz and Sprumont.Marks-Sultan et al., 2016). There are twenty-six cantons in the Swiss Confederation, each with its own constitution, legislature, executive, and judiciary. Where health is concerned, Switzerland has a two-tier system built on the federal constitution and cantonal legislation, giving cantons the largest share of responsibilities. Cantons implement regulations in areas where the Federal State has adopted laws, but can also adopt their own health policies, laws, and regulations.
6.2.1 Overview of the Core Principles of the Right to Health
The right to health means that States must establish ethically and culturally suitable policies that address local needs, as well as plans for measures and resources for promotion of national health according to their individual capacities. Two principles that are key to this right are non-discrimination and equality. States must recognize and provide for groups having specific needs and generally facing health-related challenges. And since Deaf people are particularly vulnerable in terms of health, access to care is a major topic of discussion in the local Deaf community.
At the international level, the right to health was first recognized in the Preamble of the Constitution of the WHO in 1946 (WHO, 1948). Because this treaty is binding for Switzerland as a Member State, the country should ensure maximum health for its population by protecting and promoting appropriate measures. According to WHO (WHO, 1948, Preambule, §2), health is “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity”. The right to health is also recognized in the Universal Declaration of Human Rights of 1948, in Article 25, which states that “[e]veryone has the right to a standard of living adequate for the health and well-being of himself and of his family, including food, clothing, housing and medical care […]”. States must take active measures to assure suitable quality of life for all their citizens. Adequate health is also defined in Article 12 of the International Covenant on Economic, Social and Cultural Rights (ICESC, 1976) as “ … the right of everyone to the enjoyment of the highest attainable standard of physical and mental health.” This treaty was ratified by Switzerland in 1992.
At a national level, the 1999 Swiss Constitution is the most significant legal document. It views the right to health as a duty of the State (articles 41 and 118) and prohibits discrimination on the basis of origin, race, sex, age, language, social situation, way of life, religious, philosophical or political beliefs, or psychological and mental deficiencies (article 8). On January 1, 2004, the Disability Equality Act came into force at the federal level, stating that all disabled persons have the same right to barrier-free access to social services (article 2). However, French-speaking Switzerland currently lacks sign language interpreters. Thus the Deaf community’s access to health services can be enhanced by tools that can effectively bridge the gap between the need for language services in healthcare contexts and their actual availability.
The United Nations has developed three key documents that frame the understanding and promotion of accessibility: the World Programme of Action concerning Disabled Persons; the United Nations Standard Rules on the Equalization of Opportunities for Persons with Disabilities; and the Convention on the Rights of Persons with Disabilities. These require governments and the international community to ensure equal rights and opportunities for persons with disabilities. Particular attention is paid to access – first, to information and communication, and second, to public services such as healthcare. Of the three documents listed above, the Convention on the Rights of Persons with Disabilities (UNCRPD) (United Nations, 2006) is particularly important. In this document, which entered into force in 2008, the international community undertook a political and legal commitment to include people with disabilities in all aspects of society. Article 25 of the UNCRPD states that “persons with disabilities have the right to enjoy the highest attainable standard of health and that States Parties have to take all appropriate measures to ensure access for persons with disabilities to health services.” Switzerland ratified the UNCRPD on 15 April 2014, thus making the same commitment.
As we rely more on technologies, the impetus increases to build tools functional for Deaf sign language users to enhance their equal access to healthcare. Well-designed tools should have the potential to improve users’ quality of life and independence. Accordingly, we have reviewed the legal framework and the potential impact of technology on Deaf people’s access to healthcare to explain our decision to create a version of BabelDr for LSF-CH. We go on now to a general review of existing tools developed for hospitals, followed by a description of the BabelDr application for LSF-CH.
6.3 Sign Language Translation Tools for Hospitals
With increased mobility worldwide, an increasing number of patients require translation services in healthcare settings. In order to respond to this demand, many medical translation applications for mobile phones have been developed (Reference Khander, Farag and Chen.Khander et al., 2018). However, resources for sign languages are still lacking, despite progress in machine translation and in automatic sign language processing, both in sign language recognition and sign language animation (Reference Bragg, Koller and BellardBragg et al., 2019; Reference Ebling, Johnson, Wolfe, Antona and StephanidisEbling, 2017; Reference Papastratis, Chatzikonstantinou, Konstantinidis and KonstantinidisPapastratis et al., 2021; Reference Sáfár, Glauert, Pfau, Steinbach and WollSáfár and Glauert, 2012).
Sites do exist that provide resources and popular explanations related to medical terminology for Deaf communities, such as PisourdFootnote 1 in Switzerland or World Health SignFootnote 2 (Spanish/Italian project). One famous project for the collection of medical terminology was developed in Australian Sign Language (Auslan): the Medical Signbank projectFootnote 3. In view of a perceived lack of health and medicine vocabulary, this project conducted linguistic research among Auslan users. The collected signs were made available on the Signbank site. Interpreters and the Deaf community could then provide feedback concerning them (Reference Johnston and Napier.Johnston and Napier, 2010).
Some text-to-sign phraselators using human-recorded videos also exist, but the number of sentences they translate is limited, and translation is often into American Sign Language (ASL) only. Moreover, the methodology used to produce sign language videos is often unclear, and information is often lacking concerning extension of the systems to other content or sharing of resources. In Europe, TraducMedFootnote 4, a French tool first used for the medical care of migrants, offers text-to-sign translation in LSF, to be used in medical practices or hospitals. More recently, at the Department of General Practice of the University Medical Center Göttingen, a multilingual application informing about COVID-19 vaccinations has been developed, aimed at vaccination candidates with limited proficiency in the local language. There are thirty-nine target languages, including a German Sign Language (DGS) module equipped with a set of videos (Reference Noack, Schäning and Müller.Noack et al., 2022).
Hybrid medical tools have also been developed for the medical sector that combine human-recorded videos and avatar generation. For example, in Romania, the Carol Davila University of Medicine and Pharmacy in Bucharest and the Faculty of Sociology in Pitesti, in collaboration with teachers of Romanian Sign Language (LSR), implemented a corpus of video recordings in LSR related to oral health. The corpus could also be augmented through online editing using the JASigning animated avatar (Reference Chiriac, Tivadar, Podoleanu, Balas, Jain and KovačevićChiriac et al., 2016). The team then worked on a comparative study of the two characters – human and avatar – with consideration of their advantages and disadvantages (Reference Chiriac, Tivadar and Podoleanu.Chiriac et al., 2015). A recent tool for medical use was built in the Netherlands (Reference 174Roelofsen, Esselink, Mende-Gillings, Smeijers, Mitkov, Sosoni, Giguère, Murgolo and DeyselRoelofsen et al., 2021), where the research group conceived a modular text-to-sign system that allows healthcare professionals interactively translate from written Dutch or English into Dutch Sign Language (NGT). The doctor enters a sentence or series of words in the search bar. He/she then chooses the closest match found within a database of written sentences. The system then shows prepared signed videos, some using recordings of human interpreters and some using a synthetic sign language module employing the JASigning virtual avatar. The team selected the human-recorded or the avatar videos according to the complexity or topic of the questions. (For example, videos with human interpreters were used when questions on ethical issues were asked.) (Figure 6.1).
Prototype of SignLab, Dutch Medical Application: human recording (left); avatar generation (right)
There are also tools that use sign language recognition to allow patients to answer. HospiSign, a Turkish interactive translation platform, was developed to assist Deaf patients in the hospital reception area on a daily basis (Reference Süzgün, Özdemir and CamgözSüzgün et al., 2015). At the reception terminal, the HospiSign interface displays a written question with its corresponding videoFootnote 5. The lower part of the screen displays various possible answers. The Deaf patient or his or her caregiver reproduces the corresponding signs. (Sign recognition is handled by a Microsoft Kinect v2 sensor, which has been configured to follow and recognize the hand movements of users when they respond (Reference Süzgün, Özdemir and CamgözSüzgün et al., 2015, p. 82).) He or she then moves on to the next question. Once all the questions have been answered, a summary report is provided to the doctor.
The last decade has seen growing interest in sign language translation systems that seek to empower Deaf people in hospital settings. However, prototypes remain limited to very specific domains. They offer only text interaction and provide no generic tools for developing sign language resources (Reference Albrecht, Jungnickel and von Jan.Albrecht et al., 2015). It is sometimes unclear what methodology has been used to translate written sentences into sign language, and videos are rarely shared with the research community. And for LSF-CH in particular, there is no specific open-source tool for the medical sector apart from the above-mentioned Pisourd website. Clearly, then, new tools are sorely needed that can address the needs of Deaf patients and their caregivers to increase access to hospitals. In the following section, we present our approach to creation of speech-to-sign fixed-phrase translators with the BabelDr platform and to production of sharable resources in LSF.
6.4 BabelDr for Swiss-French Sign Language
In contrast with other fixed-phrase translators, BabelDr aims for easy portability to new domains and coverage: it should be possible to continually add new content. Adapting BabelDr to sign language therefore requires flexible solutions. Human videos recorded by sign language interpreters/translators are known to be ideal, but they pose many technical problems. In particular, they cannot be generated productively and cannot be changed once recorded. We therefore decided to combine human and avatar videos, as suggested by Reference 174Roelofsen, Esselink, Mende-Gillings, Smeijers, Mitkov, Sosoni, Giguère, Murgolo and DeyselRoelofsen et al. (2021).
The translation of BabelDr content was carried out in two steps. First, a reference corpus consisting of video translations with human translators was created for a subset of sentences, in order to develop reference translations for many terms and typical structures. The first set of recorded videos was then annotated and used to develop a larger corpus of videos generated with a virtual human (an avatar). In the following sections, we explain (1) the methodology used to film the human translations and (2) how the avatar version was generated and integrated into BabelDr to develop a flexible speech-to-sign translator.
6.4.1 Recording Translations with Deaf Experts
We used a community participatory approach to translate a first set of sentences from written French into LSF-CH. The team working on the translation is comprised of a Deaf nurse and two Deaf sign language specialists (both working as sign language teachers and translators). Also in the exchange group is a doctor currently doing a specialization in Switzerland who – together with a translation researcher (a co-author of this work) – organizes sign language courses in hospitals in French-speaking Switzerland. As of March 2022, 2,661 medical questions have been translated and validated (1,552 for the hospital reception unit, 1,063 in the field of abdominal pain, and 46 specific to COVID-19).
Three main challenges were encountered by the translation work group. 1) The translation of medical jargon. The use of specific terminology in the medical context is well known to be a source of serious misunderstandings in medicine (e.g., Reference Ong, Haes, Hoos and LammesOng et al., 1995). Translation problems are frequent even for widely used languages (Reference MajorMajor, 2012). In the Deaf community, the problem is compounded: specific medical terms are rarely used (see also Major et al. 2013) and often there is no sign that would be universally accepted by the community – as in the case of “spleen” or “bile ducts”. 2) The translation of proper names, such as the names of medication like Dafalgan®, for which there is no specific sign. Translators consider that using the manual alphabet to translate these names would cause excessive eye strain for Deaf people watching the video.3) The recording medium. Videos require a switch to a two-dimensional presentation, which is especially challenging when sentences must be partially signed on the signer’s back.
Solutions that our translation work group found for these challenges were: 1) the use of paraphrases when a word was unambiguous and its meaning could be paraphrased with general concepts considered easily understandable by the patients; 2) the use of subtitles when the meaning of a word was ambiguous and a short paraphrasis was not possible; 3) the use of images to clarify the meaning of a word (e.g., the image of a specific part of the body to ensure that the Deaf patient understands the intended location, or the image of a specific medicine). Table 6.1 displays a few sentences and the strategies employed to translate specific terminology.
| Sample sentences | Strategies employed to translate | |
|---|---|---|
| are you allergic to aspirin? | Subtitle | |
| are you allergic to codeine? | ||
| have you taken anticoagulants today? | Paraphrasis | AGAINST-BLOOD-MASS |
| have you stopped taking antiarrhythmics? | MEDECINE-FOR-HEART-RHYTHM-STABILITY | |
| have you taken any treatment for osteoporosis today? | BONE-INSIDE-BRITTLE | |
| do you also have pain in the upper left side of your back? | Image | |
| do your shoulder blades also hurt? | ||
To record our translations in real time, we used the LiteDevTools online platform developed at the University of Geneva (https://regulus.unige.ch/litedevtools/client/#/login), designed to facilitate the recording of oral/video translations (Reference Strasly, Sebaï and RigotStrasly et al., 2018).
6.4.2 Virtual Avatar Generation
One way to generate virtual animation is to rely heavily on humans throughout the whole production process, exploiting motion capture and/or animation by hand. This technique can make the final rendering quite realistic. Another way is to use automatic sign language processing (Reference Ebling, Johnson, Wolfe, Antona and StephanidisEbling et al., 2017). For BabelDr, we opted for translation via a fully synthesized avatar developed by the School of Computing Sciences at the University of East Anglia (United Kingdom) – the JASigning avatar. The system’s main version (Reference Ebling and Glauert.Ebling and Glauert, 2013) is freely available for research purposes and provides several virtual characters. It was developed in the context of the European Union-funded ViSiCAST (Reference BanghamBangham, 2000), eSIGN (Reference Zwitserlood, Verlinden, Ros and van der Schoot.Zwitserlood et al., 2005) and DictaSign (Reference Efthimiou, Fotinea, Hanke, Miesenberger, Karshmer, Penaz and ZaglerEfthimiou et al., 2012) projects. In the context of BabelDr, the avatar Françoise was selected for its realism, ethnic neutrality, and expressiveness.
The JASigning avatar is based upon a notation system called G-SiGML (Gestural Signing Gesture Markup Language), which enables the transcription of sign language gestures (Reference Elliott, Glauert, Jennings and Kennaway.Elliott et al., 2004). The application uses XML to encode the features of individual signs using the Hamburg Notation System for Sign Languages (HNS) (Reference Prillwitz, Leven and ZienertPrillwitz et al., 1989). HNS describes the physical form of the signs (Figure 6.2) and has been developed to support transcription of the hands’ activity: handshape, orientation, location, and movement (Table 6.2). G-SiGML also allows researchers to represent non-manual features: facial expressions, body expressions, and gesture mouthing.
HNS description of NURSE in LSF-CH: gloss (top); image with cross movement represented by arrows (middle); HamNoSys (HNS) notation (bottom)
| Dominant hand (right hand) | Non-dominant hand (left hand) | |||
|---|---|---|---|---|
| Symbol | Description | Symbol | Description | |
| Handshape |  | The hand forms a closed fist with the thumb extended. |  | The hand forms a closed fist. |
| Orientation |  | The extension of the index finger is oriented to the signer’s left and the orientation of the palm to the left and down the axis of rotation. |  | The extension of the index finger is directed toward the front of the body, to the right of the signer and the orientation of the palm downwards. |
| Location |  | The right thumb touches the signer’s right shoulder. |  | The hand is located in front of the lower abdomen. |
| Movement |  | The hand moves down, forward, up slightly on the outside left of the signer and then moves to the right. |  | No movement is made. |
To facilitate the production of the G-SiGML code, we developed the SigLa platformFootnote 6. Its aim is to generate G-SiGML from two main resources: 1) a lexicon that associates individual signs (named with glosses) with their HNS representation; and 2) a synchronous context-free grammar that productively maps source sentences into their corresponding sign tables.
Sign tables (Table 6.3, below) are intermediate representations of signed utterances (Reference RaynerRayner, 2016). They specify a sequence of glosses (the manual signs defined via HNS) and associate them with non-manual features. The tables consist of eight rows that represent the parallel channels of signed output. The first row, GLOSS, specifies the sequence of glosses. The second row, APERTURE, refers to the degree of openness of the eyes, for example, ClosedLeft or Small. The third row, BODY, describes the movement of the body, for example, RotateLeft or TiltRight. The fourth row, EYEBROWS, describes the movement of the eyebrows, for example, Up or LeftUp. The fifth row, GAZE, indicates where the signer is looking, for example, Down or LeftUp. The sixth row, HEAD, describes the movement of the head, for example, TurnRight or TiltedBack. The seventh row, SHOULDERS, refers to the movement of the shoulders, for example, RaiseLeft, HunchBothForward. The eighth and last row, MOUTHING, describes the movement of the lips, cheeks, tongue, or teeth. The associated grammar describes the link between these sign tables and generated sentences, using variables (terminal and non-terminal symbols) as described in Rayner, 2016.
| Gloss | BE_1SG | DOCTOR | SPECIALIST | HEART |
|---|---|---|---|---|
| Aperture | Wide | Wide | Wide | Wide |
| Body | Straight | Straight | Straight | TiltBack |
| Eyebrows | Neutral | Neutral | Up | Neutral |
| Gaze | Neutral | Right | Neutral | Down |
| Head | Neutral | TurnRight | Neutral | Neutral |
| Shoulders | Neutral | Neutral | Neutral | Neutral |
| Mouthing | null | medsa | spesialis | kO: |
Once the lexicon and the synchronous grammar are ready, they can be uploaded to the SigLa platform and compiled. The SigLa platform then produces the G-SiGML code for sentences as generated by the grammar, or for specific rules only. During generation, each element of the sign table is mapped to the corresponding G-SiGML element. SigLa also aims to facilitate rule development: while storing all necessary resources and enabling the grammar developer to produce the signed animation for a sentence, it also enables modification of the corresponding grammar rule if necessary. Table 6.4 shows the resulting G-SiGML representation of the sign NURSE in LSF-CH.
6.4.3 Speech2sign Version of BabelDr
When new sentences are added in the BabelDr application, their G-SiGML code is generated with the SigLa platform. They are imported into BabelDr with the metadata and stored with other translation resources (SL human videos and written translations in other languages) so that they can be played directly in real time with JASigning in the BabelDr application. The two versions of BabelDr (with human videos and avatar generation) are accessible onlineFootnote 7, along with non-signed languages. Figure 6.3 shows the doctor and patient views for both versions.
Doctor and patient view of BabelDr with human and avatar videos
As of March 2022, the glossary consists of 608 HamNoSys entries: 370 nouns, 82 verbs/actions, 57 adjectives, 36 adverbs, 19 transfer signsFootnote 8, 15 pronouns, 8 prepositions, 5 forms of punctuations, 3 interjections, and 3 conjunctions. The grammar consists of 438 rules with 121 non- terminal and 381 terminal symbols, and can generate G-SiGML code for 1,234,828 sentences. For compliance with the FAIR principles (Findable, Accessible, Interoperable, and Reusable), the parallel corpus of human and avatar videos is now fully available on the Yareta Swiss repository in .webm and .mp4 formats (for human recordings) and G-SiGML files (for the avatar-based version).Footnote 9
6.5 Qualitative Evaluation on the Perception of Avatars and Human Videos
How do Deaf people in French-speaking European countries perceive the use of human and avatar videos in the BabelDr context? To find out, we created an online questionnaire (Reference Bouillon, David, Strasly and Spechbach.Bouillon et al., 2021). The survey, launched in four languages (LSF, LSF-CH, French Belgian Sign Language [LSFB] and written French) was implemented through LimeSurvey, an accessible online survey platform. A “snowball” sampling method was used to recruit respondents, who were given six weeks to participate. Thirty-two questions were divided into six sections on the following themes: 1) background of the videos; 2) additional images added to clarify content; 3) subtitles; 4) screen format and size; 5) advertisements and logos displayed on the screen; and 6) perception of the use of three-dimensional avatars.
The questionnaire, written in French, was made accessible through videos in LSF, LSFB and LSF-CH, all made by Deaf professionals who are native speakers of these sign languages. Each theme was introduced by a short video summarizing the topic covered. Responses were limited to “yes/no” or multiple-choice (Reference Haug, Herman and Woll.Haug et al., 2015). The questionnaire is available on the Research outputs tab of the Swiss Centre for Barrier-free CommunicationFootnote 10.
We focus here on results concerning the appreciation of virtual characters. Four questions were asked:
Past studies have determined that Deaf people may have problems in understanding the signs performed by an avatar (Reference Huenerfauth, Zhao, Gu and Allbeck.Huenerfauth et al., 2008; Reference Kipp, Heloir and Nguyen.Kipp et al., 2011). While our current work may also have demonstrated a certain preference for traditional human interpretation (Question 2: 64 percent; N=16/28), we also find that the use of automatic technologies associated with virtual characters is not without interest for the target audience. Considering the abstention rate (Question 1: 9.7 percent; N=3/31) and the negative rate (Question 1: 12.9 percent; N=4/31), our study shows that in fact most Deaf respondents (Question 1: 77.4 percent; N=24/31) do find the video information provided by a virtual character useful in the medical context (Figure 6.4).
Results of our online survey. “Question 1. Do you consider that videos with avatars can be useful?” (N=31)
Concerning the display of avatars on the screen, a major number of respondents [Question 3: 64 percent; N=18/28] prefer the signer to be shown front-on only. The options proposing multiple perspectives on the same screen or a 45° left/right perspective were almost unanimously rejected (Figure 6.5).
Results of our online survey. “Question 3. To better understand the signer, which video would you prefer?” (N=28)
Respondents could also leave personal comments if they wished. One of our participants was particularly conscious of the possibility of customizing avatars (i.e., with respect to physical appearance, age, sex and origin), so that no patients need feel uneasy or excluded:
(…) l’avatar est intéressant, car on peut choisir enfant, homme, femme, blanc, noir, etc. selon l’éthique auquel certains peuvent s’identifier sans aucune discrimination.
(…) the avatar is interesting because we can choose a child, a man, a woman, white, black, etc. according to the ethics [ethnicity would be the correct word here, but in French our Deaf respondent wrote “ethics”] of people and the group they identify with without discrimination [our translation]
Some of our respondents provided suggestions for improving the avatar. In particular, they suggested that we emphasize some movements in order to make sentences more understandable. For example, they suggested adding shrugs of the shoulders, frowning eyebrows, and a more intense look in order to make the avatar more understandable:
Pour toute interrogation, on hausse les épaules quand il s’agit des questions.
(…) je trouve qu’il manque des expressions faciales pour montrer que c’est une question.
Every time we ask a question, we shrug our shoulders (…) I find facial expressions are missing to show that we are asking a question [our translation].
6.6 Conclusions and Future Work
This chapter examines the difficulties faced by Deaf people in gaining access to healthcare. In contrast to the few other existing translation tools for the Deaf, the BabeldDr project aims to create a pipeline for productive development of quality sign language resources and to make those resources available to doctors for diagnosis via a flexible speech-enabled fixed-phrase translator, or phraselator. To produce the signed videos used in BabelDr, we have developed innovative platforms directly linked with BabelDr, including LiteDevTool and SigLa. Our results include several corpora, including (1) a reference corpus of human LSF videos for medical questions and instructions and (2) a large artificial corpus of SiGML representations.
An initial questionnaire for Deaf people concerning their perception of avatars showed that 77 percent of respondents found the information conveyed by the avatars useful in this context, although they preferred human videos. Even if avatars are far from perfect, this technology seems promising for emergency situations and for the production of sign language video corpora.
This research is pioneering in our field. To our knowledge, it is the first automatic speech translation system with sign language used in hospitals for diagnosis. The system will soon be evaluated on diagnostic tasks with the Deaf population. That evaluation will also allow us to validate the quality of the human and avatar videos and to compare patients’ satisfaction level related to the diagnostic task. In another study, we showed that Arabic and Albanian patients have less confidence in their doctor when a speech translation system uses speech synthesis instead of human recording for translated output (Reference Gerlach, Bouillon, Troqe, Mallem and SpechbachGerlach et al., 2023). It would be interesting to replicate that study for Deaf patients with human and avatar videos.
The SigLa platform will soon be made available to researchers. Experiments will be conducted to involve Deaf people in the development of the SigLa grammar and the validation of the SiGml code. We also intend to evaluate the effort needed to port the grammar to another Swiss sign language, for example, to Italian sign language or another closely related sign language, for example, LSFB. To facilitate the translation of medical sentences into real-human videos, we plan to employ Deaf people currently studying in a new training program developed at our Faculty, a Diploma in Advanced Studies (DAS) for Deaf translators.
6.7 Acknowledgments
The authors would like to thank the translation work group, especially Tanya Sebaï, Adrien Pelletier, Sonia Tedjani, Evelyne Rigot and Valentin Marti; Jonathan Mutal for the development of the SigLa platform and to thank Johanna Gerlach for her comments on this chapter.
Open access