Augmented reality, supernatural talents & medicine’s future

I spoke on a panel at the Digital Sports medicine Conference earlier this month, where one of the main themes was to picture the medical world in the year 2037. 

A small number of us studied the most recent research on the clinical applications of virtual and augmented reality in preparation for the event and analyzed in detail the current status of the field. 

I must admit that I was blown away by how far augmented reality (AR) has come in the last eighteen months for medical applications. So much so that I don’t believe AR will make a significant impact on the field until 2037. 

In fact, I believe that by the end of this decade, surgeons, radiologists, and other medical professionals will be using augmented reality on a regular basis. Many of us will visit the family doctor in the early 2030s and be tested by a specialist wearing AR glasses.

Doctors will gain superpowers thanks to augmented reality.

I’m referring to superhero imaging abilities for medical photos, patient records, and other clinical staff. The costs of these new features are already low, and they will continue to fall when augmented reality technology is consumed in greater quantities in the coming few years.

Doctors will be able to gaze right into a patient’s body and view proof of trauma or disease at the exact area in their body where it lies using augmented reality. 

Of course, equipment like CT and MRI scanning allows doctors to see beyond the skin, but they must now study these images on flat displays and visualize how they relate to the patient on the table. 

This mental shift is impressive, but it takes time and effort, and it isn’t nearly as useful as if scientists could simply look into the human body. 

The superpower of x-ray vision has become a reality thanks to augmented reality goggles and new techniques for recording 3D medical images of a patient’s physical body. An experimental emergency room was tested with the ability to capture whole-body CT scans of trauma patients and allow the medical team, all wearing AR headsets, to peer into the patient on the exam table and see the trauma in the actual area where it resides, according to an impressive study from Teikyo University School of Medicine in Japan. 

This saved time, reduced distraction and eliminated the need for mental transformations by allowing the team to discuss the injuries and arrange treatment without having to switch back and forth between flat panels.

In other words, AR technology pulls medical information off the screen and inserts them in 3D space, ideally aligned with the patient’s body. I believe that such a feature will be widely embraced in medical applications because it is so natural and intuitive. 

In fact, I predict doctors in the early 2030s will regard the traditional method of gazing back and forth at flat screens as complex and unsophisticated.  

Beyond x-ray vision, augmented reality technology will provide doctors with helpful content that will be superimposed onto (and inside) the patient’s body to assist them with clinical activities. 

Surgeons doing a delicate treatment, for example, will be given navigational signals projected on the patient in real time, indicating the precise place where interventions must be conducted with precision. 

The goal is to improve accuracy while reducing mental effort and speeding up the process. From minimally invasive treatments like surgery and histology to freehand surgical operations like inserting orthopedic implants, surgery has a lot of possibilities.

Since the key technologies were first invented, AR researchers have aimed to create augmented surgery. In truth, it dates back to the early 1990s, when the Air Force Research Laboratory built the first AR system. 

The purpose of the study was to demonstrate how augmented reality could improve human dexterity in precision procedures like surgery. As someone who was part of the early work, I can attest to the incredible development the discipline has achieved in the decades since. 

Consider this: in 1992, we used human subjects to evaluate the first AR system, which required users to place metal pegs between holes spaced two feet apart in order to see if virtual overlays could improve physical performance. 

Now, thirty years later, a team from Johns Hopkins, Thomas Jefferson University Hospital, and Washington University used AR to help with the placement of metal screws with precision under two millimeters on 28 patients. 

The screw-placement technology achieved such accurate registration between the real patient and the virtual overlays, according to recent research, that surgeons scored 98 percent on standard performance indicators. 

As accuracy has reached clinically practical levels, we can expect augmented reality to touch all parts of medicine in the future. Furthermore, big developments are in the works that will make using AR in medical contexts faster and easier.

As previously said, correct registration of the real and virtual worlds is the most difficult task for any precision augmented reality application. This now entails the patient is marked with physical markers in medicine, which requires time and effort.  

Researchers from Imperial College London and the University of Pisa have tested a “markerless” AR system for surgeons that employ cameras and artificial intelligence to exactly connect the real and virtual worlds. 

Their process was quicker and less expensive, but it was less precise. However, this is still early in the game; in the coming years, this technology will make AR-assisted surgery possible without the use of expensive markers.

Similarly, camera-based registration systems will push augmented reality out of highly regulated areas such as operating rooms and into a broader spectrum of medical applications. Indeed, I believe that by 2030, general practitioners will routinely see patients who have used augmented reality headsets.

This leads me to the next superpower I predict doctors will have in the near future: the capacity to see into the past. This is because doctors will be able to utilize AR headsets to acquire 3D photos of their patients and then view them aligned with their bodies. 

For example, a clinician may examine a patient with AR glasses to monitor the healing process of a skin lesion, interactively glancing back and forth in time to compare the current view with what the lesion looked like during previous visits. 

Overall, researchers’ development of medical applications of virtual and augmented reality is excellent and intriguing, with important implications for medical education and practice. 

“The beneficial significance of AR and VR in the upskilling of health care cannot be understated,” says Dr. Stefano Bini of the UCSF Department of Orthopaedic Surgery. 

I agree with Dr. Bini and would go even further since I believe augmented reality will have a far broader impact on the workforce than healthcare. 

After all, x-ray vision, navigational cues, dexterity support, and the capacity to gaze back in time will be valuable in anything from constructing and auto repair to engineering, manufacturing, agriculture, and, of course, education. With AR glasses being developed by some of the world’s most powerful companies, such as Microsoft and Apple, as well as Meta, Google, Magic Leap, HTC, and Snap, these superpowers will almost certainly take money over the next 5 – 10 years, enhancing all aspects of our daily lives.   

Louis Rosenberg, Ph.D., is the CEO and principal scientist of Unanimous AI, and his work in VR, AR, and AI has earned him over 300 patents.