AR glasses help surgeons operate on tumors

15 Nov 2017

NIR cameras capture fluorescence and produce a 3D reconstruction of the affected lymph node - without requiring radio-isotopes.

Malignant tumors often form metastases that spread to other parts of the body via the lymphatic system. High surgical skills are required to identify the precise location of the affected lymph nodes, enabling them to be completely removed. Researchers at the Fraunhofer IGD (Institute for Computer Graphics) researchers have developed a navigation aid that simplifies such interventions.

An augmented reality (AR) system called 3D-ARILE superimposes a virtual image of the exact position of the lymph nodes using data glasses. It will be presented to the public for the first time at the Medica trade fair, in Düsseldorf, Germany, from November 13 to 16.

According to the German Federal Statistical Office, the number of patients treated for skin cancer in German hospitals has grown significantly in recent years. Malignant melanoma, or black skin cancer, is the diagnosis that most people fear. Cancer cells can, for example, be transported by the lymph fluid to the lymph nodes. This leads to the growth of secondary tumors, or metastases. The first lymph nodes to be affected are referred to as sentinel lymph nodes.

Locating sentinel lymph nodes

Despite advances in medical science, it is still difficult for doctors to determine the precise location of sentinel lymph nodes during surgery and to check that the affected lymph nodes have been completely removed. 3D-ARILE, developed by researchers at the Fraunhofer IGD, provides doctors with a navigation aid for lymph node removal. The Clinic for Dermatology at Essen University Hospital and Trivisio Prototyping collaborated with the Fraunhofer researchers on this project.

The novel augmented reality system, based on the use of data glasses, helps surgeons locate lymph nodes with the aid of virtual markers. What makes these AR glasses special is that they work in combination with powerful medical navigation software, a stereoscopic, near infrared camera system, and indocyanine green fluorescent dye. “To make the affected lymph nodes visible, the fluorescent dye is injected into the patient in the direct vicinity of the primary tumor.

It then spreads along the lymphatic pathways and collects in the sentinel lymph nodes,” said Fraunhofer IGD research scientist Dr Stefan Wesarg. The dye fluoresces when exposed to infrared light, in this case generated by infrared LEDs. NIR cameras capture the fluorescence and produce a 3D reconstruction of the affected lymph node.

This virtual image showing their exact position is superimposed in real time and can be viewed by the surgeon through the data glasses. The necessary software was developed by the research team in Darmstadt. “In our case, the malig- nant tissue is marked in green. In this way, the surgeon can verify that every last trace has been removed,” said Wesarg.

Fluorescent dye instead of radioactive nanocolloids

Until now, doctors used technetium-99m as a medical tracer. By replacing this radioisotope with ICG fluorescent dye, patients can be treated without risk of radiation side-effects. Time is another key factor. The level of radiation emitted by lymph nodes containing the radioactive marker is very low.

This made it necessary to use scintillation (or gamma) cameras that required around 30 minutes to capture the images needed to determine the precise location of a lymph node. The data glasses, by contrast, display the site of the affected lymph node instantly.

This greatly relieves the stress on the OR staff, who no longer need to watch an additional monitor and compare the image on the screen with that on the camera’s display. “As a result, the doctor can focus entirely on the patient and carry out surgeries without hassle,” says Wesarg.

Comfortable glasses

Another advantage of the AR glasses is that they are light and extremely comfortable to wear, as dermatologists at Essen University Hospital have confirmed after carrying out numerous tests. The partners who developed the system remained in close contact throughout the project, enabling them to design a navigation aid optimized to the surgeon’s needs.

The augmented reality system comprises hardware and software components. The hardware was developed by Trivisio Prototyping. In addition to the data glasses with an integrated camera and two displays, designed specifically for medical applica- tions, the company also developed two infrared and two visual cameras. These four optical devices are integrated in a cube-shaped unit suspended over the operating table.

The Fraunhofer IGD research team was responsible for developing the software. It includes an image processing system, which detects fluorescent light emitted by the lymph nodes, uses this data to calculate their 3D coordinates, and displays their position in the data glasses. The software also performs the necessary calibration of the hardware system.