Dustin Tyler chats with Brandon Prestwood.
Uniting man and machine
Starting with a precious sense of touch, Dustin Tyler’s Human Fusions Institute is bringing new sensations to people and science.
By Zachary Lewis
Technology, to Dustin Tyler, PhD ’99, is more than a tool. It’s an extension of humanity. In his mind, man and machine belong together.
That’s a view shared by his peers in the Human Fusions Institute, a multidisciplinary, advanced engineering institute formally launched this year at Case Western Reserve University. They’re working to wisely and safely usher in that vision of a connected future.
At the HFi, as it’s called, the big-picture mission is to foster a “thriving, just, connected society.” Their more immediate goal is to mechanize the sense of touch, to apply the most human of powers to the realm of robotics.
“The more I think about it, the more massive I realize this is,” said Tyler, the director and founder of the HFi and the Kent H. Smith II Professor of Biomedical Engineering at the Case School of Engineering. “This is a paradigm shift. We can be the tool. Once you get there, the possibilities are enormous.”
The shift is already well underway. Vast as the potential for haptic technology is in gaming, healthcare, and the military, the HFi – under development by Tyler since 2017 – has progressed farthest in developing advanced prosthetics, realistic artificial hands that give wearers the ability to feel what they’re holding.
Using tiny patented “nerve cuffs” that gently wrap around and establish electrode contact with nerves, Tyler and colleagues have added sensors to prosthetics that help people missing limbs feel whole again, enabling them to touch their loved ones and perform basic tasks naturally. Others in the HFi, meanwhile, are working to support the effort financially, test it in the real world, imagine other applications and ponder ethical questions.
Keith Vonderhuevel, 54, has been involved with the institute since the beginning. The Sidney, Ohio, resident and U.S. Air Force veteran lost a hand in an industrial accident in 2004. His prosthetist introduced him to the work underway in University Circle.
Vonderhuevel now has three of Tyler’s nerve cuffs in his arm, and these, connected to sensors on the prosthetic, endow him with a sense of touch. The hand itself is operated physically, by means of external sensors that respond to muscular commands.
What he’s able to do today is more than he ever imagined, and his most powerful memory is that of picking up his granddaughter with all the delicacy and nuance the task requires.
“It was just incredible,” Vonderhuevel said. “I was able to feel how hard I was grabbing her. It was the first time I felt like I wasn’t squeezing too hard.”
The life-changing potential of this technology can hardly be overstated. Entrepreneurship expert Fehmida Kapadia, Director of Innovation and Ecosystem at the HFi, notes that the sense of touch is “primal to our existence. Being deprived of this takes a massive toll, and as the world becomes more remote, we are losing this essential thing that connects us as human beings.”
Stunning as it is, the technology is advancing rapidly. Soon, Vonderhuevel and others with HFi prosthetics won’t have cables protruding from their arms or any concerns associated with open skin. Instead, all communication between nerves and prosthetic will occur wirelessly, through an implantable device akin to a pacemaker.
Without the visual clutter of wires, Tyler suspects brains will more readily accept prosthetics as natural hands and fill in more of whatever sensory data may be missing.
“This is what the brain does,” Tyler explained. “It’s an associator. We don’t have to be perfect. We just have to be on the right channels. We need to develop a signal the brain hasn’t had before.”
Leaping beyond prosthetics
Advanced prosthetics may be the most heartwarming use of what Tyler calls “neuro reality,” but they’re far from the only application. One day soon, Tyler believes, it will be common to don devices that tap the neural network and provide sensory feedback as one plays games, performs dangerous tasks, or even examines infectious patients, all from a distance.
In that not-too-distant future, bombs will be neutralized remotely, people will explore immersive environments, and robots can be controlled with enough sensitivity to complete medical procedures.
Physician and HFi member Jonathan Baskin, MD, is especially excited about that. He expects to see this technology revolutionize surgery.
A specialist in sleep apnea and an Associate Professor in the CWRU School of Medicine and the Department of Biomedical Engineering, Baskin already employs robots in some operations. But those machines are limited, he said. If they provided tactile feedback and let him distinguish bone from soft tissue, “That would be very useful,” he said. “That would take it to the next level.”
Getting to that level doesn’t require a huge leap. Once the HFi becomes fluent in the signals the body uses to communicate, signals can be coded to operate almost any device. Touch will be a universal digital language.
“What we’re trying to do is understand how the body takes in information and at what point it can’t tell the difference,” said Melissa Schmitt, a Clinical Research Nurse at CWRU and the U.S. Department of Veterans Affairs who regularly sees HFi subjects. “We’re trying to understand more of that language, to be as natural as possible.”
Some of that language is already well understood. In August 2020, using tools and software he helped develop, Luis Mesias ’20, an electrical engineer and a doctoral student at Case, stood in Cleveland and linked to a robotic arm in California. He gently picked up what he later learned was a banana 3,000 miles away.
Sensors in a glove fed him tactile data about the size, shape, and firmness of the unknown distant object and allowed him to control the robotic hand to pick it up without crushing it. The next step, said Michael Fu, MS’06, PhD ’11, an Assistant Professor of Electrical Engineering and Computer Science, is to add touch to all five fingers and other parts of the hand.
Mesias also wrote a game in which the player, wearing a virtual-reality headset and sensor-equipped glove, attempts to tap but not pop bubbles. Without touch, the game is difficult. Once the player can feel the bubbles, however, it’s relatively easy.
That sense of touch is becoming a more lucrative pursuit.
In April, the Human Fusions Institute was named one of 38 semifinalists in a $10 million global competition called the Avatar Xprize. Sponsored by the Xprize Foundation, the contest challenges researchers to develop “an avatar system that will transport human’s sense, actions and presence to a remote location in real time.”
Tyler believes the banana demonstration, among others, puts HFI and its technology in a strong position for a prize that will be awarded in 2022.
The institute is supported in part by grants and a portion of a $20 million fund for biomedical engineering advances that late alumnus Robert Aiken ’52 and his wife, Brenda, committed in 2017. Meanwhile, Tyler increasingly collaborates with experts in related fields.
The hand out West, for instance, was built by Veronica Santos, director of the Biomechatronics Laboratory at the University of California, Los Angeles. She said she and Tyler have moved in the same spheres for a long time but their work only merged at this point of translating sensation into signal and giving robots the sense of touch.
“To see it come together when we wanted was really exciting,” Santos said. “I think our expertise is quite complementary.”
How to create “Amazing”
The collaboration between CWRU and UCLA – just one of several partner universities affiliated with the HFi – raises two other issues vital to the “thriving” and “just” future HFi imagines: privacy and network efficiency.
Remotely operating haptic technology—also known as 3D touch–depends on the safe, rapid transmission of vast amounts of personal data over long distances. The HFi therefore includes technicians establishing robust, high-speed networks and academics pondering the legal, ethical, and social ramifications of the work.
“We’re looking to bring the entire university into this,” Tyler said. “We’re not just throwing tech out there and letting someone else deal with it. If you pull in other people from the start, you’re creating something amazing.”
Others on campus are also striving to crack the body’s electric code and transform lives. Abidemi “Bolu” Ajiboye, PhD, an Associate Professor of Biomedical Engineering at the Case School of Engineering, has long been working parallel to Tyler, deciphering neural signals. His work does not involve robots. Rather, his aim is to re-animate existing limbs disconnected from the brain by spinal cord or other injuries. He creates devices that re-route signals around the problem.
Ajiboye has had success restoring function where it appeared to be lost forever. His work also has received significant attention from other researchers and was showcased in the 2020 documentary, “I Am Human.”
Between his research and Tyler’s, Ajiboye said, there are “natural synergies, and we’re exploring those in real time. We’re right at that inflection point where things could start moving really quickly.”
Venkataramanan “Ragu” Balakrishnan, Dean of the Case School of Engineering, sees a Case and Cleveland expertise converging.
“The Human Fusions Institute exemplifies the ideal for research and innovation in the Case School of Engineering,” he said “Their work draws on the university’s foundational excellence in biomedical engineering and leverages our unique partnerships in the Cleveland medical ecosystem.
From that core strength, the researchers have moved beyond traditional disciplinary boundaries to explore new possibilities, he said.
Through that exploration, lives are being changed.
Brandon Prestwood lost his left hand nine years ago when his arm got ensnared in a conveyor belt in an industrial laundry. The mangled limb was amputated just below the elbow.
On a July afternoon, he was the center of attention at the Cleveland Functional Electrical Stimulation Center in the basement of the Cleveland VA Medical Center, where Tyler has a lab. In 2017, members of Tyler’s team implanted nerve cuffs in the residual muscles, allowing the 46 year old to feel sensation in an artificial arm. Prestwood had come up from Hickory, North Carolina, for further testing and diagnosis.
“Obviously, the sensation is incredible,” he said slowly, almost overcome with emotion. “I got to hold my wife’s hand and give her a real hug for the first time (since the accident). But it makes me complete. It makes me feel whole again. This program makes me useful to society again.”
The HFi appears ready to do more. Its prosthetics are the best possible proof of concept, and it’s gaining new partners and supporters every day. Meanwhile, efforts to establish safe networks and translate the electrical signals of the body into a universally accessible code are proceeding apace. Once those are available, new uses for remote and virtual touch technology are almost certain to blossom.
The HFi still needs to secure more funding, obtain federal approval for new equipment, and gather members scattered around campus into a physical headquarters with offices, labs, and multipurpose space. Tyler remains unsure where their campus address will be. He also aspires to develop coursework and one day offer a degree specialization or certificate program.
With others around the globe exploring related areas of human-tech interaction, there’s a certain sense of urgency. Besides that, humanity can’t wait, team members believe. Many of the challenges people like Keith Vonderhuevel face are ones the HFi is uniquely suited to solve.
“We need to move faster,” Kapadia said. “At the end of the day, that’s why most of us are doing what we’re doing: to change the world in some way.”
Zachary Lewis is a former award-winning reporter for The Plain Dealer and a Cleveland freelance writer. To comment on this story and to ask questions, please email firstname.lastname@example.org.
Graduate student researcher Leah Marie Roldan preps Keith Vonderhuevel for a test of the sensory powers of his advanced prosthetic developed by researchers at the Human Fusions Institute.
Two models of touch-enabled robotic hands rest in a lab affiliated with the Human Fusions Institute. More advanced equipment is in development.
Keith Vonderhuevel demonstrates the use of an advanced prosthetic that endows him with the sense of touch.
Biomedical engineer Dustin Tyler, PhD ’99, believes brains will one day accept prosthetics as natural hands