How digital twins are helping people with motor neurone disease speak
Tech companies, a charity and academics have collaborated to create AI-powered avatars that are helping people with motor neurone disease hold natural conversations
An initiative by a UK-based charity, supported by technology companies and universities, has developed an artificial intelligence (AI)-powered digital twin that allows people with communications disabilities to speak in a natural way.
The technology, known as VoxAI, represents a step-change from the computer-assisted voice used by late physicist Stephen Hawking, one of the first well-known public figures with motor neurone disease (MND).
The Scott-Morgan Foundation was set up by its founder, roboticist Peter Scott-Morgan, to apply engineering principles to disability after he was diagnosed with MND.
A five-year project led by the trust has developed an AI-powered platform that is helping people with MND, also known as amyotrophic lateral sclerosis (ALS), to communicate in a natural way despite their disabilities.
It was developed by the foundation’s chief technologist, Bernard Muller, who is paralysed with MND and has learned to write code using eye-tracking technology.
The platform brings together AI technologies to create photo-realistic avatars that move in a natural way, with natural facial expressions, and can reproduce the voice of the person using it. It is able to listen to the conversation and offer disabled people a choice of three answers that they could select based on its understanding of the person.
One of the people testing the technology, Leah Stavenhagen, for example, worked as a consultant at McKinsey before she developed MND. The AI she uses has been trained on a book she wrote, along with 30 interviews in English and French.
LaVonne Roberts, CEO of the Scott-Morgan Foundation, told Computer Weekly that while people did not mind waiting to hear what Stephen Hawking had to say, delays in communication usually cause problems for both the speaker and the listener.
“When you have someone that is having to spell something out laboriously, they are fatiguing their eyes, which has been shown to further progression of MND, so we are trying to protect from that,” she said.
“The other thing that happens is people start giving much shorter answers because they don’t have the time to stay in a conversation,” added Roberts. “And, honestly, you end up with awkward pauses.”
The Scott-Morgan Foundation, which demonstrated the technology today at an AI Summit in New York, plans to make the software available free of charge, so that it can be used by as many people as possible. It will also offer a subscription version for more advanced features.
Many off-the-shelf computers and tablets now come with workable eye-tracking, and tracking devices provided by the NHS may also be able to use the technology, said Roberts.
“The idea was to democratise the technology by putting it on the web, giving the license keys, so that people have their voice back again,” she said.
More than 100 million people in the world who live with conditions that severely limit speech – including people recovering from a stroke, or living with cerebral palsy, a traumatic brain injury or non-verbal autism – could benefit from the technology.
The foundation plans to start a two-year trial of the platform, which will track some 20 participants using the technology, led by Mexican university Tecnológico de Monterrey, which will evaluate its impact.
It is also developing a simplified platform that could be used by people who do not have access to Wi-Fi.
Gil Perry, CEO of D-ID, which creates digital avatars for businesses, contributed to the project after the company helped a few people with MND/ALS in ways they found life-changing.
His company joined the project with the Scott-Morgan Foundation about two years ago, after meeting with Roberts. “I saw that LaVonne has the vision and can connect all the dots together, because she has a group of people who just sleep and dream that vision day and night,” said Perry.
The company has improved its technology so that it can create an avatar that shows facial expressions, even for someone whose condition means they are at an advanced stage of immobility.
Roberts said that one of the breakthrough moments came after a mother told the foundation that, although the technology was good, “You just didn’t capture my daughter’s smile”. That sparked work to make the avatars more lifelike. “I remember Erin’s mother crying when she saw Erin on a video, and she was like, ‘That’s her smile’,” she said. “And I knew we were onto something.”
Muller, who architected the platform, said that his avatar not only makes him visible, but also “present”. “When someone sees my avatar smile or shows concern, they are seeing me, not a disability,” he added. “That changes everything about how I connect with the world.”
