AI for Social Good – Google AI Blog

Google’s AI for Social Good crew consists of researchers, engineers, volunteers, and others with a shared give attention to optimistic social impression. Our mission is to reveal AI’s societal profit by enabling real-world worth, with initiatives spanning work in public well being, accessibility, disaster response, local weather and vitality, and nature and society. We consider that the easiest way to drive optimistic change in underserved communities is by partnering with change-makers and the organizations they serve.

In this weblog publish we focus on work carried out by Project Euphonia, a crew inside AI for Social Good, that goals to enhance automated speech recognition (ASR) for individuals with disordered speech. For individuals with typical speech, an ASR mannequin’s phrase error charge (WER) could be lower than 10%. But for individuals with disordered speech patterns, reminiscent of stuttering, dysarthria and apraxia, the WER might attain 50% and even 90% relying on the etiology and severity. To assist handle this drawback, we labored with greater than 1,000 individuals to accumulate over 1,000 hours of disordered speech samples and used the info to indicate that ASR personalization is a viable avenue for bridging the efficiency hole for customers with disordered speech. We’ve proven that personalization could be profitable with as little as 3-4 minutes of coaching speech utilizing layer freezing methods.

This work led to the event of Project Relate for anybody with atypical speech who may gain advantage from a customized speech mannequin. Built in partnership with Google’s Speech crew, Project Relate permits individuals who discover it laborious to be understood by different individuals and know-how to coach their very own fashions. People can use these customized fashions to speak extra successfully and achieve extra independence. To make ASR extra accessible and usable, we describe how we fine-tuned Google’s Universal Speech Model (USM) to raised perceive disordered speech out of the field, with out personalization, to be used with digital assistant applied sciences, dictation apps, and in conversations.

Addressing the challenges

Working intently with Project Relate customers, it turned clear that customized fashions could be very helpful, however for a lot of customers, recording dozens or a whole lot of examples could be difficult. In addition, the customized fashions didn’t all the time carry out effectively in freeform dialog.

To handle these challenges, Euphonia’s analysis efforts have been specializing in speaker unbiased ASR (SI-ASR) to make fashions work higher out of the field for individuals with disordered speech in order that no further coaching is important.

Prompted Speech dataset for SI-ASR

The first step in constructing a sturdy SI-ASR mannequin was to create consultant dataset splits. We created the Prompted Speech dataset by splitting the Euphonia corpus into prepare, validation and take a look at parts, whereas guaranteeing that every cut up spanned a variety of speech impairment severity and underlying etiology and that no audio system or phrases appeared in a number of splits. The coaching portion consists of over 950k speech utterances from over 1,000 audio system with disordered speech. The take a look at set incorporates round 5,700 utterances from over 350 audio system. Speech-language pathologists manually reviewed the entire utterances within the take a look at set for transcription accuracy and audio high quality.

Real Conversation take a look at set

Unprompted or conversational speech differs from prompted speech in a number of methods. In dialog, individuals communicate quicker and enunciate much less. They repeat phrases, restore misspoken phrases, and use a extra expansive vocabulary that’s particular and private to themselves and their neighborhood. To enhance a mannequin for this use case, we created the Real Conversation take a look at set to benchmark efficiency.

The Real Conversation take a look at set was created with the assistance of trusted testers who recorded themselves talking throughout conversations. The audio was reviewed, any personally identifiable data (PII) was eliminated, after which that knowledge was transcribed by speech-language pathologists. The Real Conversation take a look at set incorporates over 1,500 utterances from 29 audio system.

Adapting USM to disordered speech

We then tuned USM on the coaching cut up of the Euphonia Prompted Speech set to enhance its efficiency on disordered speech. Instead of fine-tuning the complete mannequin, our tuning was based mostly on residual adapters, a parameter-efficient tuning method that provides tunable bottleneck layers as residuals between the transformer layers. Only these layers are tuned, whereas the remainder of the mannequin weights are untouched. We have previously proven that this method works very effectively to adapt ASR fashions to disordered speech. Residual adapters have been solely added to the encoder layers, and the bottleneck dimension was set to 64.

Results

To consider the tailored USM, we in contrast it to older ASR fashions utilizing the 2 take a look at units described above. For every take a look at, we evaluate tailored USM to the pre-USM mannequin greatest suited to that activity: (1) For quick prompted speech, we evaluate to Google’s manufacturing ASR mannequin optimized for brief kind ASR; (2) for longer Real Conversation speech, we evaluate to a mannequin skilled for lengthy kind ASR. USM enhancements over pre-USM fashions could be defined by USM’s relative measurement enhance, 120M to 2B parameters, and different enhancements mentioned within the USM weblog publish.

Model phrase error charges (WER) for every take a look at set (decrease is best).

We see that the USM tailored with disordered speech considerably outperforms the opposite fashions. The tailored USM’s WER on Real Conversation is 37% higher than the pre-USM mannequin, and on the Prompted Speech take a look at set, the tailored USM performs 53% higher.

These findings recommend that the tailored USM is considerably extra usable for an finish consumer with disordered speech. We can reveal this enchancment by taking a look at transcripts of Real Conversation take a look at set recordings from a trusted tester of Euphonia and Project Relate (see under).

A comparability of the Pre-USM and tailored USM transcripts revealed some key benefits:

• The first instance exhibits that Adapted USM is best at recognizing disordered speech patterns. The baseline misses key phrases like “XBox” and “controller” which are essential for a listener to know what they’re attempting to say.
• The second instance is an effective instance of how deletions are a main concern with ASR fashions that aren’t skilled with disordered speech. Though the baseline mannequin did transcribe a portion accurately, a big a part of the utterance was not transcribed, dropping the speaker’s meant message.

Conclusion

We consider that this work is a crucial step in the direction of making speech recognition extra accessible to individuals with disordered speech. We are persevering with to work on enhancing the efficiency of our fashions. With the fast developments in ASR, we intention to make sure individuals with disordered speech profit as effectively.

Acknowledgements

Key contributors to this challenge embrace Fadi Biadsy, Michael Brenner, Julie Cattiau, Richard Cave, Amy Chung-Yu Chou, Dotan Emanuel, Jordan Green, Rus Heywood, Pan-Pan Jiang, Anton Kast, Marilyn Ladewig, Bob MacDonald, Philip Nelson, Katie Seaver, Joel Shor, Jimmy Tobin, Katrin Tomanek, and Subhashini Venugopalan. We gratefully acknowledge the help Project Euphonia obtained from members of the USM analysis crew together with Yu Zhang, Wei Han, Nanxin Chen, and lots of others. Most importantly, we wished to say an enormous thanks to the two,200+ individuals who recorded speech samples and the numerous advocacy teams who helped us join with these individuals.

¹Audio quantity has been adjusted for ease of listening, however the unique information can be extra in step with these utilized in coaching and would have pauses, silences, variable quantity, and so on. ^↩