New MIT imaging system senses through murky waters

For remotely operated underwater vehicles, cloudy and turbulent waters are often inaccessible. When vehicles settle on the seafloor or dig through a sandbed, they can create clouds of sediment that make it difficult for onboard cameras to see through. But a new underwater mapping technique developed by engineers at MIT and the Woods Hole Oceanographic Institution (WHOI, Woods Hole, Mass.) may enable vehicles to perceive through low-visibility waters. 

The method fuses visual images from optical cameras with acoustic data from sonar sensors. This combination enables a vehicle to quickly map the general shape of its surroundings using sonar, even in low-visibility waters.

MIT says the technique is “akin to pairing a dolphin’s echolocation with a sea turtle’s close-range vision to see and navigate through murky water, in real-time.” 

The team is further improving the technique, which they have named Sonar-MASt3R. They envision that the mapping method could safely guide underwater vehicles through murky environments for a range of applications. 

“We hope that this work enables us to do more operations in those challenging, low-visibility environments, and helps provide more coverage in areas that are difficult to operate in today,” said Amy Phung, a graduate student in MIT’s Department of Aeronautics and Astronautics, who led the work. 

Phung presented a paper detailing Sonar-MASt3R  at the IEEE International Conference on Robotics and Automation (ICRA), in Vienna, last week. Co-author is Richard Camilli, senior scientist of applied ocean physics and engineering at WHOI. 

‘Opti-acoustic fusion’

To get the best of both sonar and optical modes, scientists have looked to combine the two in a new approach known as “opti-acoustic fusion.” In a handful of prior works, research groups have merged sonar and optical data in mapping techniques that are mostly geared toward object recognition and reconstructing workplace environments. Most techniques require time to sync and process the data and therefore do not work in real-time, while only a few can map an environment in 3D.

The MIT-WHOI Joint Program team was motivated, in part, by challenges in safely recovering unexploded underwater mines. “There can be old explosives in areas that make it unsafe for ships to be in, and the ability to get rid of those safely is best done by robotics,” said Camilli. “But a lot of these explosives are set in surf zone environments where visibility adds to the challenge of doing this safely.”

The Sonar-MASt3R builds on an existing technique, MASt3R, which was developed by researchers in France. MASt3R is an image-matching algorithm that is trained to take in visual images of the same scene and quickly estimate the relative depth of each pixel in the scene. In this way, MASt3R can generate a 3D map of the environment in real-time, based on a camera’s 2D images. But that system was limited by problems with its scaling limitations.

In the new work, Phung and Camilli used sonar data to correct MASt3R’s scaling and generate precise 3D maps of underwater environments. Even in murky water, the method’s sonar-corrected map would enable a vehicle to know the precise location of objects, and therefore how far to safely move in for a closer inspection, which the vehicle could then do using conventional optical cameras.

The team tested Sonar-MASt3R in experiments with a tank that they filled with water, sediment, and a variety of objects such as a small boulder, a coffee mug, and a packing crate. For each experimental run, they first carried out a sweep trajectory, in which a robotic arm slowly swept from one side to the other to capture sonar and visual data.

With its first sweep, Sonar-MASt3R quickly creates a coarse sonar-based map of the shapes and contours of the tank and its objects. The coarse map is then used to record close-up camera images of the objects, which are used to improve the map resolution.

The researchers tested their new approach underwater, at eight different levels of turbidity, which they created by stirring up the tank’s sediment. Compared with other opti-acoustic fusion approaches, Sonar-MASt3R generated more accurate 3D maps and resolved smaller, centimeter-scale details, and in cloudier conditions.

The team plans to test the approach in natural underwater conditions, where they suspect that the mapping task should be more straightforward. “In a tank, it’s like an echo chamber,” said Camilli. “It’s like trying to do this in a funhouse mirror setting where you get all these distortions and reverberations and ghost images that really complicates the processing. If you put it in the real world, it should be easier.”