How did you get started with this research?
My career at Columbia began around 16 years ago and throughout that time my main focus has been looking at optical networks and optical interconnects for various scales of computing systems. Everything from looking at on-chip interconnects to datacentres and long-haul metro area networks but my interest and focus on HPC has been going on for at least a decade.
There was a DARPA report looking at the projections for building an exascale system back in the early 2000’s and then the report came out in 2008. All the way back then I was very keen and involved in making use of photonics in these systems.
Back then was really the first time that we hit on how power consumption was going to become such a huge factor in scaling. Right around that time I would say mid-2000’s was also around the time that in the integrated photonics world there was also a corner turned.
Silicon is not a very good photonic material simply because it does not have a direct end gap like some semiconductors. This means that it is not a very good material to make lasers and detectors. On the other hand, silicon is extremely prevalent, we have enormous capabilities in CMOS fabrication and so forth. And so we had the idea of moving photonics into the integrated circuit world – but still, with this barrier of not being able to do anything too useful in silicon was challenging.
What happened in the mid-2000s was coincidentally with the 90 nm CMOS fabrication node. That was a turning point because, when you are able to fabricate with 10’s of nm precision, the optical devices that you can fabricate with that material have less loss. The optical wavelength is very large, integrated photonics is much larger than the transistor but because the technology was developed for CMOS to be able to lithographically make billions of transistors.
Where you were able to do tens of nanometre or a hundred nanometre-type precision we could create optical waveguides which are the equivalent of optical wires with enough precision that the walls are smooth enough that the losses are small.
All of a sudden, because of a technology that was originally developed for CMOS and the semiconductor industry, we are now able to make optical things in the CMOS process with pretty good characteristics such as low loss – and this was one of the things that really propelled the field allowing us to take integrated optics right into the silicon platform.
Fast-forward to now, and there has been really tremendous progress – so today we are able to make all the components of photonic links and this includes 50 GHz data modulators, 50 GHz receivers, nanosecond scale switches all kinds of structures and we can do wave division multiplexing into these interconnect structures. The capabilities are just enormous.
What are the advantages of using optical communications?
The advantage of the interconnect technology is that because of the low losses. We know from larger scale systems like fibre optics networks that optics is a great communication technology but the reason for this is twofold. One is that we can send the light over very long distances before we need to amplify it because the losses in the optical transmission waveguide can be very low – that helps with the power consumption. In electronics, we have to regenerate the bits over much smaller distances.
We can send optical light over long distances without having to spend energy- without having to reamplify. It is independent of the data rate – whereas in electronics the higher the frequency, the higher the data rate the more power that is being expended. In the optical world, the optical fibre carries 10 GB, 100GB or a TB and the energy that is dissipated is independent of the frequency and the distance.
In HPC the distances are much shorter, we are talking about communicating from chip to chip but the point is that in the optical domain it's really all the same. Communicating across the chip and across the board or between racks is done with the same technology because I can send the light over these distances without having to regenerate it. Conversely, in the electronic world, you have to use a different technology with every distance.
All those steps expend energy and that is one of the reasons that, as soon as you get off the chip and off the board, your bandwidth diminishes substantially. In the optical domain, it is all the same. In other words, I can have very high bandwidth densities for short distances and the same exact technology for long distance. This opens up the possibility of having a much more flat architecture in terms of bandwidth communications.
The other beautiful thing about optical communications is that fundamentally it is bosons and not fermions, so they do not interfere with each other. This means that we can put multiple lines of communication in a single wire over multiple wavelengths and they don’t interfere with each other. In a single waveguide I can send 16 or 32 channels, but maybe one day soon this will be increased to 128.
What is the potential for the technology?
Obviously, there has been quite a bit of progress both in terms of the technology development and the integration densities but the other part that happened and is just now coming to fruition is the ecosystem. If we fast-forward to today and we think about commercially including silicon photonics within these systems, what does it take to commercialise? It is the same ecosystem that needed to be developed for the semiconductor industry.
We need to develop everything from automated design tools for the layout of the chips, foundries that fabricate the integrated photonics similar to CMOS fabrication processes, we need packaging, and this is just now coming to fruition worldwide.
In the US, for example, this institute started in 2016 it is called the American Institute for Manufacturing Photonics (AIM Photonics) and this was a huge investment of $600 million from the US government and the state of New York.
It builds on a 300mm CMOS fabrication facility that is dedicated to photonics. That is the core of the institute, but there are other components there including all the other major components of the ecosystem design tools and a packing facility that is currently being constructed.
It is a challenging gap between the invention of new technology, as good as it may be, and deploying it. There are similar initiatives happening in Europe and in the Far East (primarily in Singapore).
In HPC we want to increase performance but we want to do it within a power envelope. It is trying to increase performance while still maintaining this power envelope.
For datacentres (DC) I think there is less emphasis on power consumption so it is really about performance and at the DC market it is really driven by cost and so at this point in time the optics are still more expensive than electronics so DC use the optics at the locations in the DC where higher bandwidth or longer distances are needed.
It is kind of interesting because it is kind of the opposite to CMOS. While CMOS fabrication is very expensive, or at least the infrastructure to do that is very expensive in optics the fabrication is actually not that expensive relatively speaking. It is really the packaging whereas in CMOS the packaging is very cheap.
The inclusion of the laser source in the package is not a completely solved standard problem at this point. The packaging challenge is how you have the chip that has both an optical and an electronics I/O all together in a commercial, low-cost product.
Lots of groups and companies are working at it but it is not a done deal. There are multiple approaches at the moment in silicon photonics. Do you integrate the laser onto a silicon platform? Do you use an external laser?
All those questions are being dealt with but I think we are not too far from converging to a more standard solution. We are maybe two years away from a pretty standardised solution.
Keren Bergman will deliver her keynote presentation at ISC High Performance 2018, in Frankfurt, Germany at the end of June.
