IBM claims co-packaged optics ‘breakthrough’ for data centers

11 Dec 2024

Polymer optical waveguide design achieves 5x reduction in power consumption when scaling GenAI.

Researchers at IBM have developed a new process for co-packaged optics (CPO) to enable connectivity within data centers at the speed of light to complement existing short reach electrical wires.

The company this week stated that “by designing and assembling the first publicly announced successful polymer optical waveguide (PWG) to power this technology, IBM researchers have shown how CPO will redefine the way the computing industry transmits high-bandwidth data between chips, circuit boards, and servers.”

Today, fiber optic technology carries data at high speeds across long distances, managing nearly all the world's commerce and communications traffic with light instead of electricity.

Although data centers use fiber optics for their external communications networks, racks in data centers still predominantly run communications on copper-based electrical wires. These wires connect GPU accelerators that may spend more than half of their time idle, waiting for data from other devices in a large, distributed training process which can incur significant expense and energy.

IBM researchers have demonstrated a way to bring optics’ speed and capacity inside data centers. In a technical pre-print paper on arXiv, IBM presents its new CPO prototype module that can enable high-speed optical connectivity. This technology could significantly increase the bandwidth of data center communications, minimizing GPU downtime while drastically accelerating AI processing.

This research innovation, as described, would enable:

• Lower costs for scaling generative AI through a more than 5x power reduction in energy consumption compared to mid-range electrical interconnects1, while extending the length of data center interconnect cables from one to hundreds of meters.
• Faster AI model training, enabling developers to train a Large Language Model (LLM) up to five times faster with CPO than with conventional electrical wiring. CPO could reduce the time it takes to train a standard LLM from three months to three weeks, with performance gains increasing by using larger models and more GPUs (figures based on training a 70 billion-parameter LLM using industry-standard GPUs and interconnects.).
• Dramatically increased energy efficiency for data centers, saving the energy equivalent of 5,000 U.S. homes’ annual power consumption per AI model trained (Figures based on training a large LLM (such as GPT-4) using industry-standard GPUs and interconnects.).

‘Future-proofing’

“As generative AI demands more energy and processing power, the data center must evolve – and co-packaged optics can make these data centers future-proof,” said Dario Gil, SVP and Director of Research at IBM. “With this breakthrough, tomorrow's chips will communicate much like how fiber optics cables carry data in and out of data centers, ushering in a new era of faster, more sustainable communications that can handle the AI workloads of the future.”

In recent years, advances in chip technology have densely packed transistors onto a chip; for example, IBM’s 2 nm node chip technology can contain more than 50 billion transistors. CPO technology is intended to scale the interconnection density between accelerators by enabling chipmakers to add optical pathways connecting chips on an electronic module beyond the limits of today's electrical pathways.

IBM’s paper outlines how these new high bandwidth density optical structures, coupled with transmitting multiple wavelengths per optical channel, have the potential to boost bandwidth between chips as much as 80 times compared to electrical connections.