 |
18 Aug 2015
Freeform prisms, LED arrays and optical tracking systems are among the photonics technologies advancing the capability of 'augmented reality' applications.
Augmented reality (AR) is finding increasing use in a variety of commercial, civil and military applications – and photonics technology is playing a key role. But exactly what kinds of components are needed, and how are they being used? Andrew Williams reports:
Surgical assistance
One of the major prospective applications for AR is medicine. A team of researchers at Kettering University in Michigan, US, is working on a project to develop innovative software tools that simulate surgery to aid physicians with pre-operative planning and intra-operative navigation, as well as training for robotic and minimally invasive surgery.
For many years a key trend has been the evolution of “key-hole” laparoscopic procedures, helping to minimize post-operative trauma for the patient. However, this typically involves complex equipment and setups, which make the procedure more challenging for surgeons – something that has prompted the development and integration of computer-based tools and robotic aids.
Already, a range of optical technologies are being used - including software that allows surgeons to visualize an individual patient's anatomy in 3D, and so-called “surgical navigators” that can be integrated in the operating room, providing additional modes and capabilities such as a real-time 3D view while surgical tools are in position.
Project leader Giuseppe Turini, an assistant professor of computer science at Kettering, says there are several different ways to use optics and laser technologies in this field. These include stereo laparoscopic camera systems, auto-stereoscopic 3D displays, a variety of depth sensors, marker-based optical tracking systems, and camera systems based on structured light.
Turini’s team has used optical tracking systems to detect the position of surgical instruments in real-time during surgery, and recently won a major research instrumentation program grant to acquire a marker-based optical tracking system called the NDI 3D Investigator.
Ergonomics
He and colleague Justin Young have since used that tracking system to conduct ergonomics studies of touch-free gestural human-computer interactions, but there is the potential to do much more. “In the future I may use this system, or a consumer depth camera, for example the Microsoft Kinect, to integrate real-time tracking in my virtual simulators," Turini says.
Each of the devices used by the team features different optical technologies. Marker-based tracking systems typically use infrared sensors to determine the 3D position via time-of-flight data. Alternatively, depth cameras can use different approaches - for example, the first version of the Kinect system uses structured light to retrieve a depth image, while version two employs a time-of-flight approach.
“Obviously, in order to use optics and laser technologies in a surgical scenario we need to consider that occlusions may happen during the intervention,” Turini says. “This is always an issue with optical systems, and usually it is solved using redundant cameras and sensors or addressing the gaps in the marker detection in software.”
He believes that the ongoing advances in depth-sensing technology will lead to major innovations in computer-assisted surgery. In future, that could enable seamless and robust tracking of all the surgical instruments and other tools in the operating room, without imposing any constraints on either the patient or the surgical team.
“[A]t the moment all the surgical simulators I have developed can be used only in the research field,” Turini points out. “In order to develop a commercial product for the medical and surgical field, a significant investment is required to cover all the expenses related to safety [and] regulations.”
Freeform prisms aid Marines’ training
Another field of application with obvious potential is the military, and US-based SA Photonics has recently developed what it calls a low-cost augmented reality system (LARS) for use by the US Marines in outdoor training exercises.
LARS is a transparent binocular head-mounted display (HMD) that provides high-resolution images and a wide field of view, and which is designed to minimize any obscuration of peripheral vision while maximizing situational awareness. Each eye sees 1920 x 1200 resolution, full-color video imagery that is overlaid with the wearer’s own view - similar to a pilot looking through an aircraft head-up display (HUD), but with a wider field of view.
Although LARS was initially designed to be fixed to a helmet, Ben Mall, director of business development for vision products at SA Photonics, explains that it can also be “soft-mounted” on a headset using a standard, commercially available night vision goggle (NVG) mechanism.
This means that the device can be easily flipped up to a secure position when not in use. LARS also features a semi-transparent, “snap-down” visor to control light to the user's eyes in bright conditions, as well as adjustments for fore/aft and vertical positioning, and inter-pupillary distance.
That adjustment terminology refers to the need for the optics to be at exactly the correct distance from the pupils, and since everybody has a slightly different-shaped head (including where their pupils are in relation to where the headset rests on their head), those adjustments need to be made when the HMD is first worn.
The optical system inside LARS is based on a freeform prism component, developed to combine a wide horizontal binocular field of view with full-color 1920 x 1200 pixel resolution for both eyes.
Low weight; large FOV
Mall says that the optical relay and combiner systems that are more commonly used in such displays rely on polarization effects, and typically result in an extremely bulky and heavy binocular assembly, with limited transparency and more difficult adjustment to the individual user.
But because LARS is based on a freeform prism with a corrector element optical system (see diagram), it can combine a wide field of view and higher transmission with a much lower weight - crucial for military applications.
“This prism provides very good performance in terms of modulation transfer function, exit pupil and eye relief while maintaining a small form factor,” Mall said. “The result is a small and lightweight optic. When integrated with a high-resolution microdisplay it is comparable to wearing standard binocular NVGs. The user sees the ‘real world scene’ overlaid with video and symbols with minimal obscuration.”
So far, LARS, which was developed under a Small Business Innovation Research (SBIR) contract from the Office of Naval Research (ONR), has been used in both field and simulated training environments.
Mall says that SA Photonics has future plans to develop HMDs that could provide binocular fields of view in excess of 140 degrees, combined with image resolutions greater than 5000 x 1200 color pixels.
LED arrays to brighten AR displays
Another example of photonics technology pushing forward AR comes from France, where researchers at CEA-LETI have recently developed very-high-brightness LED arrays for use in next-generation HUD applications.
Ludovic Poupinet, head of the CEA-LETI optics and photonics department, told optics.org: “One of our photonic labs, which specializes in display technologies and display systems, noticed that the main limitation in displays for augmented reality was the brightness of the display. So we decided to study the application of an LED matrix to transparent head-mounted displays.”
More specifically, this has involved the development of novel patterning technology for gallium nitride (GaN) LED epiwafers, to produce micro arrays of LEDs with a 10 µm pitch (or less), and fewer process steps to create an LED matrix with a high fill-factor.
“The disruption comes from the fact that we had to find a wafer-level technology to fabricate millions of micron-size pixels. In this work we benefited from the synergies between our display lab and our lighting lab,” Poupinet explains.
Improving on Google Glass
The main applications targeted by the CEA-LETI team with this new technology are transparent displays such as multimedia glasses. Poupinet and colleagues are confident that such devices represent the future of the human/digital content interface, despite the lukewarm impact so far of products like Google Glass. They believe that their novel LED matrix will help to overcome the limitations of those devices, which have yet to make much commercial headway.
“Our main challenge now is to develop color displays,” Poupinet said. “We are studying different alternatives for color emission, from thin, pixelized color conversion to direct color emission by the pixels themselves. The main challenges are to provide glasses with high transparency, a large field of view, and good visual comfort.”
He added: “What is or was acceptable for military use of head mounted displays is not [necessarily] acceptable for everyday life applications.
“The key purpose of the glasses are first to let our hands be free (unlike with smart phones) and second to let us interact with digital content [within] a large field of view. To combine a large field of view with a small form factor for the optical system is a real challenge.”
He points out that this is one of the objectives of the European project HIDO, in which CEA-LETI is also involved. “At the user interface level, we want to interact naturally with the digital content,” Poupinet said. “Everything must be intuitive with no learning curve. A 3D imaging device and the computational power to analyze gestures and position digital content correctly depending on head and eye positions are required.”
That means another challenge is power consumption, which must be kept very low to allow the use of small batteries and long periods of autonomy. Everything must be optimized to reduce power consumption and heat dissipation, which as Poupinet says is not acceptable near the head.
Currently investigating a range of different uses and commercial development opportunities for the LED matrix technology, the CEA-LETI team sees professional applications where hands-free displays can provide real advantages as the likely starting point.
About the Author
Andrew Williams is a freelance writer based in Cardiff, UK.
 |  |  |  |  |  |  |
| © 2024 SPIE Europe |
|
