A few months ago, via Zoom, I watched Guy Martin, a clinical lecturer in the surgery department at Imperial College of London, pick up some blood vessels from a patient's abdomen, flip them around, and show me where a previous surgery had rerouted the normal vasculature of the patient's kidneys.

Except what he was really holding was not viscera. In fact, he wasn't holding anything at all. It was a hologram made using 3-D CT scans.

He asked the Zoom attendees, many of whom were Michigan Medicine physicians, about various approaches to a vascular surgery for this patient. With slight finger movements, he was able to manipulate the hologram, offering us a better view of pertinent anatomy. It was all for demonstration purposes, but the implications for real-world use are vast.

"Often when we're reviewing [scans] pre-op, it's all 2-D in your own mind," says Martin. "You're trying to make a 3-D picture to figure out how you're going to get into the bit of aorta that you want to clamp." But with Hololens, the Microsoft headset Martin was using, "You can really be inside it and say, 'I can put my clamp here.'"

Enter the Matrix

This sounds like science fiction, but it's happening now.

Technology like the Hololens falls under the umbrella term "extended reality." You might see it abbreviated XR, and it basically means any situation in which real life is augmented by computer technology. That covers everything from Pokemon GO (the immensely popular 2016 smartphone game that allowed users to see fantastical, computer-animated critters in their real environment) to virtual reality headsets that immerse users in a completely computer-generated world (a la "The Matrix").

Administrative assistant Sarah Cook observes while Mark Cohen, M.D., demonstrates the Hololens headset.
The use of a holographic skeleton bypasses the need for an expensive physical model that would need storage space.
Credit: Leisa Thompson Photography

Imagine this scenario: A physician at University Hospital in Ann Arbor gets a call from a health care team in the Upper Peninsula. They have a patient in dire straits, and they need advice about this complex case. Should the physician talk them through the care of this patient or have the patient transported to the University of Michigan hospital via Survival Flight? Triage decisions must be made in seconds, with limited information. Wouldn't it be easier if the physician in Ann Arbor could just magically join the remote team and temporarily lend them her eyes, hands, and expertise?

It sounds like wishful thinking, but thanks to extended reality, it may become standard practice in the near future.

What was once the province of video games and futuristic films is now being harnessed in the medical world, and innovators at Michigan Medicine are conducting some of the first extended reality experiments in a medical setting.

"We recently put in a center grant application in collaboration with the University of Maryland and industry leaders like Microsoft, Google, Facebook, Sony, Philips, GigXR, and the FDA to become a National Science Foundation-sponsored center focused on XR applications in medicine," says Mark Cohen, M.D., professor of surgery. "We would be the first in the country in this space."

 "Our focus is to bring augmented, virtual, and mixed reality technologies to support all 19 schools and colleges [at U-M]," says Jeremy Nelson, director of the XR Initiative at the U-M Center for Academic Innovation. It's Nelson who has helped Cohen and others at Michigan Medicine get access to the technology that could radically change how health care is delivered. "The health care space has been very interested in these technologies."

Ready Player One

During the Zoom training, we never saw Martin's face. He was wearing the Hololens headset, and the virtual images he pulled up using the headset's software, such as the hologram, were mapped onto his physical space. All of this was projected to a computer screen, which we were seeing on our own screens.

From where we were sitting, we saw his hands as if they were our hands. If he looked at the mannequin patient on the OR table, so did we. In a sense, he was lending us his physical reality.

He was also lending us his virtual reality, the sophisticated holograms he could bring up with a flick of his fingers. With just one headset, dozens of physicians were able to replicate the experience of being with this "patient" and running through a pre-operative scenario.

If he were planning a complex surgery, he could also get expert advice from physicians thousands of miles away, with the click of a few buttons. And the people giving that advice would feel almost as if they were in the room with him.

Virtual Reality IRL

What happens when the patient is a real live person? It's something Michigan Medicine has already begun to explore. A couple months ago, Cohen donned PPE and a Hololens headset and entered the room of a very sick COVID-19 patient who needed to be put on ECMO (extracorporeal membrane oxygenation) life support.

"A lot of the folks that were in the Zoom call, looking at the patient from my view from the headset, had never seen ECMO, but now we were able to bring them directly to the bedside," says Cohen. "We had international experts in that headset with me, and they were talking about their experience in the UK and how it's different doing ECMO in the UK. Even our seasoned ICU faculty learned a lot from that international exchange in real-time at the bedside."

Only one set of PPE was used, and only one person risked exposure. For students who are going through medical school during a pandemic, this technology could mean access to powerful bedside training from the safety of their homes.

What Do Patients Think?

Cohen says they are careful to get patient consent through Michigan Medicine's standard process before using the Hololens. He explains to the patient that many people will be able to see and hear them, and those people might even talk to the patient through the Hololens speaker.

"Patients honestly don't mind it at all so far," he says. "One interesting observation we found is that patients seemed more at ease and comfortable talking to one person in the room with a headset on than with a whole team of providers surrounding the bedside. [There's] less anxiety than having a whole team of white coats in a room … especially for adolescent and pediatric patients."

Still, are patients weirded out by the headset? "Back in the day when we first started using tech like this, the headsets were huge and clunky, and we looked a bit 'alien' walking around with them," says Cohen. But he says some of the new headsets coming out are lower profile and "look like a stylish pair of sports glasses."

Microsoft Hololens headset. Credit: Leisa Thompson Photography

Virtual Stressors to Manage Real-World Stress

One thing that is difficult for traditional medical training to replicate is the stress and sense of urgency in the emergency department (ED). With immersive virtual reality headsets, Michigan Medicine researchers are solving that problem.

Imagine a baby is having trouble breathing. The algorithm for treatment is clear, but the distress of a real live infant might make it difficult to recall. Michelle Aebersold, Ph.D., R.N., a clinical professor at the U-M School of Nursing, has worked with the Emerging Technology Group at the U-M Duderstadt Center to develop software that allows health care workers to train for this scenario while completely immersed in an environment that looks similar to the ED where they work, and with a patient who looks a lot like a real baby.

When they put on an Oculus headset, which fully immerses the user in a virtual reality, "They see a little baby. They see the monitor. They can see the heart rate and can interact with the baby, listen to lung sounds, check for pulse," says Aebersold. She worked as a nurse at Michigan Medicine for more than 25 years, so she knows what it takes to create a realistic ED environment. "I've had nurses in the ED test it out, and one of them said, 'Oh that's a rapid infuser!'" It's not equipment that's important for the training, but it adds to the realism.

During the simulation, the baby goes into ventricular tachycardia, and the trainee must go through the treatment algorithm. "Right now, [the trainees] don't successfully resuscitate the baby, because we want them to go through the entire algorithm."

It's all an illusion, but it could help save a life in the realest of moments.

Gamifying Non-Technical Skills

It may seem like high tech would be best suited to technical training, but Michael Cole, M.D., associate professor of emergency medicine, is taking it in a different direction. "When someone's heart stops, the algorithms to treat that patient are straightforward on paper, but in the real world, clinicians and staff have to work together to get that patient's heart started again and this is where errors commonly occur" says Cole. "If we give a test on cardiac arrest care to 8 healthcare workers and had them complete it individually, the average score would be about 85%; but if we take the same 8 healthcare workers and instead asked them to sit together in a room and work collaboratively on a single test, then the score would be 100% every time."

Cole believes it's important to train healthcare students and workers in leadership, teamwork, and communication skills. That's why he has helped design a virtual reality team training module for cardiac arrest that awards points to learners based not only on the clinical choices they make but also on how well they collaborate and communicate with team members.

Using fully immersive virtual reality (VR) headsets, learners will be able to train as a team in a simulated, stressful environment. "We all put on a VR headset, then no matter where we are in the real world, we're all in the same virtual room together" says Cole. "We see our patient. The patient is in cardiac arrest." At that point, clinicians and nurses must work together to care for the patient according to the algorithm. If a physician is about to make a mistake and a nurse speaks up to stop them, the nurse would get points for good communication. And if the physician took the nurse's advice, the physician would get points as well.

The end goal is improving patient outcomes, whether you get to good outcomes by following the algorithm perfectly or by listening to a colleague who points out an error you're about to make.

We Can Do It. Does That Mean We Should?

An early adopter of XR technology in the world of medical education and real-time care delivery, Prashant Mahajan, M.D., professor of emergency medicine, of pediatric emergency medicine, and of pediatrics, as well as vice chair of the Department of Emergency Medicine, is conducting experiments that could make it possible to use the Hololens headsets in an emergency scenario similar to the one earlier in this story. Although he's excited about the implications for collaborating with remote teams, he also thinks "we need to be careful about it.

"What are the legal ramifications of this? Are there consent issues, patient confidentiality issues? Are there issues because of the technology itself? Does the technology cause a burden or increase your cognitive load now that you're using it?" As an example, Mahajan notes that electronic health records have increased the cognitive load on health care workers and contributed to burnout.

Mahajan also worries about what might happen if the technology breaks down or if broadband access is lost. Will we have created a cohort of health care professionals who are completely dependent on the technology? Mahajan compares it to the way people have become so reliant on Google maps that they no longer know how to navigate without GPS technology.

"To me, the biggest thing is, it should not worsen patient safety. It should not harm patients."

Even with all of these concerns, Mahajan is excited about how much this technology has the potential to improve patient care. "You could have multiple subspecialties managing a patient at the same time," he says. "It can be cost effective, safe, timely, and appropriate. Importantly, healthcare is no longer bounded by geography and for the first time, real-time care can be provided all across the globe."

What's on the Horizon? Just Breaking Down All the Barriers in Medicine

Virtual reality headsets have the potential to increase access to medical training, especially for health care workers who are not near a major medical center with expensive training arenas. "You could train any health care worker, on demand, remotely, without the need for physical presence," says Mahajan. "You can completely democratize medical education. And because it is overlaying your real world with digital images, where the image could be a video or 3-D hologram, it suddenly makes learning that much more meaningful and immersive."

Beyond education, the new XR technologies will improve patient care, says Cohen. The tech will increase access to specialized care, and Cohen envisions it being combined with machine learning to "better predict when diseases will flare up, and how to improve the way we follow and treat chronic diseases like heart failure, cancer, and diabetes.

"In the near future, we will be using XR to bring medical expertise from Michigan to anywhere in the world," he says. "These tools allow us, for the first time, to break down access barriers like never before, bringing expertise to the bedside; whether it is improving how we manage patients in remote areas in the U.S. or around the world."

"I think we are just scratching the surface here on what can be done with these technologies," says Nelson.

Mark Cohen, M.D., and Jeremy Nelson pose with a holographic skeleton.
Credit: Leisa Thompson Photography