AR surgeon's visor could slash operating times

06 Mar 2017

Italian-led European project merges live patient image with X-ray and anesthesia data into surgeon's head-up display.

Simultaneous view of both patient and vital data.

Employing novel photonics technology, European scientists are developing a new type of augmented reality (AR) surgical visor to improve accuracy of procedures. The system can display anesthetic and medical data while superimposing a patient’s X-ray over the body, meaning surgeons do not have to look away during an operation. The developers say this approach can reduce operation times by 20 minutes for every 3 hours.

The VOSTARS (Video Optical See-Through Augmented Reality surgical System) medical visor is a head-mounted display system, which presents anesthetic data and a patient’s vital signs including, heart rate, body temperature, blood pressure, and breathing rates, directly into the surgeon’s field of vision.

Although AR for surgical procedures has been discussed in academic and industrial research since the 1990s, a tool joining a surgeon’s natural perception with patient data has not yet been widely implemented.

However it is the use of photonics components, such as the small, high-luminous micro display, the LED optical waveguide, and a micro-projector array that can present a 2D X-ray image in front of the operator that is the key to the VOSTARS innovation.

Project coordinator Dr Vincenzo Ferrari, biomedical engineering researcher at the Department of Information Engineering, at the University of Pisa, commented, “With this ergonomic visor, we intend to provide all the information required to improve surgery. The primary goal is to reduce not just surgery times, but also the time spent under anesthetic and the cost involved in any operation.”

The VOSTARS system works by capturing what the surgeon sees from a head-mounted camera. The system then merges / augments this image of reality with the patient’s medical images, collected from CT, MRI, or 3DUS scans.

The central processor, using the most advanced registration techniques available for surgical navigation, then presents a real-time hybrid image on the visor dashboard to the surgeon.

Kinder cuts: operating times could be reduced by 11%

Hybrid

Dr Ferrari added, “However, rather than having this information on a small GPS screen, our important patient data are all integrated instantly into the surgical visor. So a clinician can move freely while still seeing the patient, the hybrid X-ray image and all of critical data all at once.”

Scientists at VOSTARS are building a new hybrid of the two existing AR headsets approaches in order to combine the features of both a video see-through (VST) system and optical see-through (OST) head-mounted display.

Fabrizio Cutolo, engineer and expert in wearable AR systems, believes neither VST nor OST alone was suitable for operating on a live patient. He said, “When operating on a real person, it became clear to us that the benefits of OST and VST systems could be combined to make a hybrid device.”

In ‘optical see-through’ (OST) systems, as in Microsoft Hololens, the user has a direct view of the surroundings with the computer-generated images superimposed on a user’s field of vision using a semi-transparent mirror.

Cutolo added, “For something as critical as an operation, the surgeon needs to have the naturalistic feel of OST, while having the fluid interaction of the VST. Therefore a brand new device needed to be made from scratch, rather than extending an existing technology.”

Now three months into the three-year project, the VOSTARS team is aiming to have a working prototype of the hybrid device ready by May 2018. Initially being trialed on a number of procedures to the head, including maxillofacial (jaws and face), neurosurgery, ENT and orthopedic surgery, a prototype is expected to be available by 2020, with mass-production by 2022.

Coordinated in Italy at Department of Information Engineering and the EndoCAS Center for Computer Assisted Surgery (Pisa University), the VOSTARS project has received a grant of €3,816,440 from Horizon 2020 via the Photonics Public Private Partnership.

Participants include: (Germany) SCOPIS GMBH, Charité – Universitätsmedizin Berlin, Sankt Gertrauden Krankenhaus GMBH, Technische Universitaet Muenchen, Pilotfish GMBH; (Italy) Alma Mater Studiorum-Universita Di Bologna, Scuola Superiore Di Studi Universitare E Di Perfezionamento Sant'Anna; (United Kingdom)VREO Innovation; (France) Optinvent, Commissariat A L’Energie Atomique et aux Energies Alternatives.