nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

This many standards is dumb: Decoding 25Gb Ethernet and beyond

Cease this silliness with haste

By Trevor Pott, 6 Feb 2017

Sysadmin Blog The 25 and 50Gb switching standards have finally been ratified. Switches from various manufacturers have been available for some time, but now there's a better than average chance they'll interoperate with one another. While more speed is generally good, the 25 and 50Gb standards will complicate things for data centre administrators by making us have to think carefully about which 100Gb switches we buy.

To understand the chaos we're stepping into with 25 and 50Gb it is important to understand the history of 10Gb Ethernet, which is actually a jumble of different standards, and not all of them actually go at 10Gb.

10Gb standards

10Gb Ethernet can be broken up into four groups. XENPAK, XPAK and X2 can be thought of as the same thing with sequentially smaller plugs. They are functionally indecomposable, and they also don't actually run at 10Gb. These use the XAUI electrical standard, which is actually four lanes of 3.125Gb.

XFP uses the XFI electrical standard, which is a single lane of 10Gb. Because of the different electrical standards getting a passive copper cable to connect between the XAUI-based ports and the XFI-based XFP isn't possible. Using a serializer/deserializer, however, it is possible.

This isn't true of the far more popular SFP+. SFP+ also allowed for direct-attach copper cabling, but because it is a "simpler" standard, with less intelligence in the transceiver and more in the PHY it can't talk to XAUI-based ports. This caused a bit of a problem.

In the early 2000s 10Gbase-CX4 emerged as the "cheap alternative" to other means of doing 10 Gbit Ethernet. It would supposedly reuse existing InfiniBand cable (though not all Infiniband cable qualified). It also got used as a stacking port in several switches. Sadly, one can't plug 10Gbase-CX4 into SFP+ without a media converter.

We can, however, plug 10Gbase-CX4 into XFP, thanks to its ability to use a serializer/deserializer. We can also plug SFP+ optical modules into an adapter that then then fits into XENPAK ports. This is again because a lot of the "smarts" are in the transceiver with XENPAK. Also, XENPAK ports are physically huge.

For added fun, SFP+ cables and transceivers emit a vendor ID and thus there exists a multitude of compatibility problems. Plugging a Cisco switch into an Arista switch, for example, will probably require you visit FibreSource and thus not use an officially supported cable. 10Gb vendors have been quite happy to abuse the SFP+ vendor ID to try to create lock in, driving up complexity and cost.

Eventually, 10GBase-T became popular, thanks in no small part to its use of dirt cheap CAT-6 cabling and the familiar RJ-45 plug. 10GBase-T does not have these issues, however, it also can't use fibre optic cables, greatly limiting the distance of cable runs.

10GBase-T ports use more power than it typically provided by SFP+ ports and thus until very recently 10Gbase-T modules to plug into SFP+ ports have been difficult to find and expensive. They are more available to today, but the run length is still limited.

Beyond 10Gb

Faster Ethernet was eventually needed. It was decided that the next steps were to be 40Gb and 100Gb. With everyone having had so much fun the last time 40Gb is actually 4x 10.3125Gb lanes. Meanwhile, 100Gb can come in either 10x 10.3125Gb lanes or 4x 25.78125Gb lanes. Because of course it can.

CFP and CFP2 are and Cisco's CPAK can present both 10 lane and 4 lane 100Gb standards. CFP 4 and QSFP+ are four-lane 40Gb standards while QSFP28 is a 4 lane 100Gb standard.

The list is rounded out by CXP, which is supposedly a 10-lane 100Gb standard but can somehow do 120Gb, with breakout cables to 3x 40Gb ports being a thing you can buy. I don't recommend looking at the price of that cable if you have a weak heart.

Interconnecting all of this is less disastrous than it was with 10Gb, but still not much fun. The industry's solution to all this confusion, lock-in and expense was to create the 25Gb and 50Gb standards. (Obligatory XKCD.) This further complicated matters.

Breakout cables allow for taking advantage of the multiple lanes these 100Gb standards are built on. Some 100Gb to 10x 10Gb cables are reasonably inexpensive, coming in at a little over $300 USD. This is not far off of what you could expect to pay for a 100Gb to 4x 25Gb cable.

Unfortunately, QSFP28 seems to be winning the optical module war. As it is only 4 lanes we only get 4x 10Gb ports when using a breakout cable. Alternately we can get 4x 25Gb ports, 2x 40Gb ports and presumable 2x 50Gb ports, once it's decided what form factor(s) 50Gb will use.

This is all really dumb

If course, this is all ridiculous. What people actually buying networking want is the most possible networking with the lowest cost and the least hassle. Enter 40Gbase-T and 25Gbase-T. These standards use Cat-8 cable and the familiar RJ-45 plug.

Where the other 100Gb standards are server and inter-switch-oriented, boasting about port density due to the ability to use breakout cables, 40Gbase-T and 25Gbase-T are likely to follow their RJ-45-based predecessors and simply focus on spamming a whole lot of "deep and cheap" ports.

With cabling being simple, not having interoperability problems and no sacrifices to the breakout cable deity required, the Base-T set of standards should ultimately win on actual shipped ports in short order. The other standards will still be used – sometimes you just need to use fibre – but none are likely to have a particularly long run as the most popular port type.

So 25 and 50 gigabit Ethernet are out. As with past 20 years of networking, however, don't expect mass adoption until the Base-T switches come down in price. For some reason, it seems like people who don't specialise in networking don't like playing guessing games with standards, vendor lock-in, or cables over $1,000. I can't fathom why.

The Register - Independent news and views for the tech community. Part of Situation Publishing