10 Gigabit Ethernet

The 10 Gigabit Ethernet or 10GbE or 10 GigE standard was first published in 2002 as IEEE Std 802.3ae-2002 and is the fastest of the Ethernet standards. It defines a version of Ethernet with a nominal data rate of 10 Gbit/s, ten times as fast as Gigabit Ethernet.

Over the years the following 802.3 standards relating to 10GbE have been published: 802.3ae-2002 (fiber -SR, -LR, -ER and -LX4 PMDs), 802.3ak-2004 (-CX4 copper twin-ax InfiniBand type cable), 802.3an-2006 (10GBASE-T copper twisted pair), 802.3ap-2007 (copper backplane -KR and -KX4 PMDs) and 802.3aq-2006 (fiber -LRM PMD with enhanced equalization).

The 802.3ae-2002 and 802.3ak-2004 amendments have been consolidated into the IEEE 802.3-2005 standard. The other amendments will be consolidated into IEEE Std 802.3-2008 which has not yet been published.

10 Gigabit Ethernet supports only full duplex links which can be connected by switches. Half Duplex operation and CSMA/CD (carrier sense multiple access with collision detect) are not supported in 10GbE.

The 10 Gigabit Ethernet standard encompasses a number of different physical layer (PHY) standards. As of 2008 10 Gigabit Ethernet is still an emerging technology with only 1 million ports shipped in 2007, and it remains to be seen which of the PHYs will gain widespread commercial acceptance. A networking device may support different PHY types by means of pluggable PHY modules.

At the time the 10 Gigabit Ethernet standard was developed there was much interest in 10GbE as a WAN transport and this led to the introduction the concept of the WAN PHY for 10GbE. This operates at a slightly slower data-rate than the LAN PHY and adds some extra encapsulation. The WAN PHY and LAN PHY are specified to share the same PMDs (Physical Medium Dependent) so 10GBASE-LR and 10GBASE-LW can use the same optics. In terms of number of ports shipped the LAN PHY greatly outsells the WAN PHY.

Optical Modules

Optical modules are not specified in 802.3 but by multi-source agreements (MSAs). The relevant MSAs for 10GbE are XENPAK, X2, XPAK, XFP and SFP+. [cite web|url=http://www.mergeoptics.de/overview.html|title=Pictures of optical modules supplied by Merge Optics]

When choosing a PHY module consideration should be given to cost, reach, media type, power consumption and size (form factor).

XENPAK was the first MSA for 10GE and has the largest form factor.

X2 and XPAK were later competing standards with smaller form factors. X2 and XPAK have not been as successful in the market as XENPAK.

XFP came after X2 and XPAK and is smaller.

The newest module standard SFP+ (see SFP transceiver) developed by the ANSI T11 fibre channel group is smaller still and lower power than XFP. It is hoped that SFP+ will enable lower cost 10GE optical modules to be produced. [cite web|url=http://www.nyquistcapital.com/2007/11/28/10gbe-and-sfp-this-time-its-different/|title=10GbE Optical Component and SFP+ Modules: This Time It's Different by Andrew Schmitt] SFP+ modules only do optical to electrical conversion, no clock and data recovery, putting a higher burden on the host's channel equalization.

Optical modules are connected to a host by either a XAUI, XFI or SFI interface.

XENPAK, X2, and XPAK modules use XAUI to connect to their hosts. XAUI (XGXS) uses a four-lane data channel and is specified in IEEE 802.3 Clause 48.

XFP modules use a XFI interface and SFP+ modules use a SFI interface. XFI and SFI use a single lane data channel and the encoding specified in IEEE 802.3 Clause 49.

Fiber (10GBASE-R)

There are two classifications for optical fiber [cite web|url=http://www.10gea.org/optical-fiber-10ge.htm|title=Optical Fiber and 10 Gigabit Ethernet white paper by the 10GEA] : single-mode (SMF) and multi-mode (MMF). In SMF light follows a single path through the fiber while in MMF it takes multiple paths resulting in differential mode delay (DMD). SMF is used for long distance communication and MMF is used for distances of less than 300 m. SMF has a narrower core (8.3 µm) which makes connecting fibers more difficult. MMF has a wider core (50 or 62.5 µm) and is more expensive than SMF. The advantage of MMF for short distances is that because of its wider core it can be driven by lower cost lasers and its connectors are cheaper and more reliable. Its disadvantage is that due to DMD it can only work over short distances. [cite web|url=http://www.corning.com/docs/opticalfiber/cn0603.pdf|title=Why choose Multimode fiber? by Corning] To distinguish SMF from MMF cables, SMF cables are usually yellow, while MMF cables are orange. [ [http://www.superwarehouse.com/10M_MMF_CABLE_SC_SC_62.5_125/09167/p/9102 Cables To Go Orange Fiber Optic Cable 33' ( 09167 ) - Fiber - Cables To Go Cable ] ]

New structured cabling installations use OM3 50 µm MMF which has no center defect. OM3 cable can carry 10GBE 300 m using low cost 10GBASE-SR optics or use 10GBASE-LX4 without a mode conditioning patch cord. See ISO 11801 and multi mode fibre.fact|date=August 2008

Older installations use FDDI grade 62.5 µm MMF which has a center defect and is harder for the 10GBE optical modules to drive [cite web|url=http://bicsi.org/Events/Conferences/Spring/2005/GeorgePRES.pdf|title=10 Gigabit Ethernet over Multimode Fiber by John George] . For -LX4 a mode conditioning patch cord is needed that adds extra cost to an installation.

10GBASE-SR

10GBASE-SR ("short range") uses 64B/66B encoding and 850 nm lasers. It is designed to support short distances over deployed multi-mode fiber cabling, it has a range of between convert|26|m and convert|82|m depending on cable type. It also supports convert|300|m operation over new, 50 μm 2000 MHz·km OM3 multi-mode fiber (MMF). The transmitter can be implemented with a VCSEL (Vertical Cavity Surface Emitting Laser) which is low cost and low power. MMF has the advantage of having lower cost connectors than SMF due to its wider core. OM3 is now the preferred choice for structured optical cabling within buildings.

10GBASE-SR delivers the lowest cost, lowest power and smallest form factor optical modules.

10GBASE-LR

10GBASE-LR is a Long Range Optical technology delivering serialized 10 gigabit Ethernet over a 1310 nm connection on single-mode fiber via IEEE 802.3 Clause 49 64B-66B Physical Coding Sublayer (PCS) using a line rate of 10.3125 .

Single-mode optical cabling is used to interconnect transceivers at a distance spaced at convert|10|km, but it can often reach distances of up to convert|25|km with no data loss.

Fabry-Perot lasers are commonly used in 10GBASE-LR optical modules. Fabry-Perot lasers are more expensive than VCSELs but their high power and focused beam allow efficient coupling into the small core of single mode fiber.

10GBASE-LR optical modules are now cheaper than 10GBASE-LX4 optical modules.

10GBASE-LRM

10GBASE-LRM, (Long Reach Multimode) also known as 802.3aq, is a 2006 standard [cite web|url=http://standards.ieee.org/cgi-bin/status?802.3aq|title=IEEE Standards Status Report for 802.3aq] which supports distances up to convert|220|m on FDDI-grade 62.5 µm multi-mode fibre (using 1310 nm) originally installed in the early 1990s for FDDI and 100BaseFX networks and convert|260|m on OM3. 10GBASE-LRM reach is not quite as far as the older 10GBASE-LX4 standard. However it is hoped that 10GBASE-LRM modules will be lower cost and lower power than 10GBASE-LX4 modules.

Because 10GBASE-LRM uses the same PCS as XFI it is a good fit for the XFP optical module whereas the 10GBASE-LX4 PCS is incompatible with XFI.

10GBASE-ER

10GBASE-ER ("extended range") supports distances up to convert|40|km over single-mode fiber (using 1550 nm).

10GBASE-ZR

Several manufacturers have introduced convert|80|km|sing=on|abbr=on range ER pluggable interfaces under the name 10GBASE-ZR. This 80 km PHY is not specified within the IEEE 802.3ae standard and manufacturers have created their own specifications based upon the 80 km PHY described in the OC-192/STM-64 SDH/SONET specifications.

The 802.3 standard will not be amended to cover the ZR PHY.

10GBASE-LX4

10GBASE-LX4 uses coarse WDM to support ranges of between convert|240|m and convert|300|m over legacy multi-mode cabling. This is achieved through the use of four separate laser sources operating at 3.125 Gbit/s in the range of 1300 nm on unique wavelengths. This standard also supports convert|10|km over SMF.

Until 2005 10GBASE-LX4 optical modules were cheaper than 10GBASE-LR optical modules.

10GBASE-LX4 is used by people who want to support both MMF and SMF with a single optical module. Because 10GBASE-LX4 uses four lasers it has a potential cost, size and power disadvantage compared to 10GBASE-LRM.

When used with legacy MMF an expensive mode conditioning patch cord is needed. The mode conditioning patch cord is a short length of SMF which connects to the MMF in such a way to move the beam away from the central defect in the legacy MMF. This is not needed with OM3.

Copper (10GBASE-X/10GBASE-T)

10G Ethernet can also run over twin-ax and cat 6a cabling as well as backplanes.

10GBASE-CX4

10GBASE-CX4 — also known by its working group name 802.3ak — transmits over 4-lanes in each direction over copper cabling similar to the variety used in InfiniBand technology. It is designed to work up to a distance of convert|15|m|abbr=on. This technology has the lowest cost per port of all 10Gb interconnects, at the expense of range. Each device capable of supporting a 10GbE module uses some MSA (Multi-Source Agreement) to provide the actual module connectivity within the device to the outside connector. XENPAK, X2, and XPAK connectors all fit into a standard MSA pinout. CX4 modules exist at least in the XENPAK, X2, and XFP variety, and possible XPAK, although the smaller size makes this configuration more difficult. Each lane of the copper carries 3.125 Gbaud of signaling bandwidth. It is the job of the 802.3ae Clause 48 protocol to manage and synchronize the data flowing between these 4 channels; this functionality is maintained in the PCS. Compare to 10GBASE-R devices, which use the Clause 49 protocol. Clause 48 uses an 8 to 10 bit conversion to accommodate better line signaling, but Clause 49 uses a 64 to 66 bit conversion for this accommodation, which leaves the actual overhead for signaling much tighter than the Clause 48.

10GBASE-CX4 offers the advantages of low power, low cost and low latency, but has a bigger form factor than SFP+.

FP+ Direct Attach

This uses a passive twin-ax cable assembly and connects directly into an SFP+ housing. It has a range of 10 m and like 10GBASE-CX4 is low power, low cost and low latency with the added advantage of having the small form factor of SFP+. SFP+ Direct Attach is expected to be the optimum solution for reaches of 10 m. [cite web|url=http://communities.intel.com/openport/blogs/server/2008/03/26/10-gigabit-ethernet-alphabet-soup-never-tasted-so-good|title=10 gigabit ethernet-alphabet soup never tasted so good]

10GBASE-KX4 and 10GBASE-KR

Backplane Ethernet — also known by its working group name 802.3ap — is used in backplane applications such as blade servers and routers/switches with upgradable line cards. 802.3ap implementations are required to operate in an environment comprising up to convert|1|m|in of copper printed circuit board with two connectors. The standard provides for two different implementations at 10Gbit/s: 10GBASE-KX4 and 10GBASE-KR. 10Gbase-KX4 uses the same physical layer coding (defined in IEEE 802.3 Clause 48) as 10GBASE-CX4. 10GBASE-KR uses the same coding (defined in IEEE 802.3 Clause 49) as 10GBASE-LR/ER/SR. The 802.3ap standard also defines an optional layer for FEC, a backplane autonegotiation protocol and link training where the receiver can set a three tap transmit equalizer.

10GBASE-T

10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 gigabit/second connections over unshielded or shielded twisted pair cables, over distances up to convert|100|m. [cite web|url=http://standards.ieee.org/cgi-bin/status?802.3an|title=IEEE Standards Status Report for 802.3an] 10GBASE-T cable infrastructure can also be used for 1000BASE-T allowing a gradual upgrade from 1000BASE-T and autonegotiation to select which speed to use. 10GBASE-T has higher latency and consumes more power than other 10 gigabit Ethernet physical layers. In 2008 10GBASE-T silicon is now available from several manufacturers [cite web|url=http://www.broadcom.com/products/Enterprise-Networking/10-Gigabit-Ethernet-Transceivers/BCM8481|title=Broadcom 10GBASE-T PHY] [cite web|url=http://www.teranetics.com/tn1010.html|title=Teranetics 10GBASE-T PHY] [cite web|url=http://www.solarflare.com/products/10xpress.htm|title=Solar Flare 10GBASE-T PHY] with claimed power dissipation of 6W and a latency approaching 1 microsecond [cite web|url=http://10gigabitethernet.typepad.com/network_stack/2008/04/10gbase-t-comes.html|title=George Zimmerman's blog] , however, these parts are not yet in production.

Connectors

10GBASE-T uses 650 MHz versions of the venerable IEC 60603-7 8P8C connectors already widely used with Ethernet.

Cables

10GBASE-T will work up to convert|55|m|abbr=on with existing Category 6 cabling. In order to allow deployment at the usual convert|100|m|abbr=on, the standard uses a new partitioned Category 6a (a.k.a "augmented Cat6") cable specification, designed to reduce crosstalk between UTP cables (formally known as "alien crosstalk").

Electrical characteristics

The 802.3an standard defines the wire-level modulation for 10GBASE-T as a Tomlinson-Harashima precoded (THP) version of pulse-amplitude modulation with 16 discrete levels (PAM-16), encoded in a two-dimensional checkerboard pattern known as DSQ128. Several proposals were considered for wire-level modulation, including PAM with 12 discrete levels (PAM-12), 10 levels (PAM-10), or 8 levels (PAM-8), both with and without Tomlinson-Harashima Precoding (THP). PAM-5 is what is used in the older 1000BASE-T gigabit Ethernet standard.

WAN PHY (10GBASE-W)

10GBASE-SW, 10GBASE-LW, 10GBASE-EW, and 10GBASE-ZW are varieties that use the WAN PHY, designed to interoperate with OC-192/STM-64 SDH/SONET equipment using a light-weight SDH/SONET frame running at 9.953 Gbit/s. WAN PHY is used when an enterprise user wishes to transport 10G Ethernet across telco SDH/SONET or previously installed WDM systems without having to directly map the Ethernet frames into SDH/SONET. The WAN PHY variants correspond at the physical layer to 10GBASE-SR, 10GBASE-LR, 10GBASE-ER and 10GBASE-ZR respectively, and hence use the same types of fiber and support the same distances. There is no WAN PHY standard corresponding to 10GBASE-LX4 and 10GBASE-CX4 since the original SONET/SDH standard requires a serial implementation. The WAN PHYs use 10GBASE-W as the PCS.

10GbE NICs

10GbE network interface cards are available from several manufacturers [cite web|url=http://www.chelsio.com/|title=Chelsio website] [cite web|url=http://www.neteffect.com/|title=Neteffect website] [cite web|url=http://www.neterion.com/|title=Neterion website] [cite web|url=http://www.netxen.com/index1.html/|title=NetXen website] [citeweb|url=http://www.silicom.co.il/|title=Silicom website] [cite web|url=http://www.myri.com/|title=Myricom website] . These plug into ordinary computer servers using PCI express, or PCI-X and connect to the LAN with a choice of PHY modules.

ee also

* Fast Ethernet
* Gigabit Ethernet
* 100 Gigabit Ethernet
* List of device bandwidths
* GG45
* TERA
* XAUI

Notes and references

External links

* Plato Networks [http://www.platonetworks.com]
* [http://www.10gea.org/Tech-whitepapers.htm 10GEA Whitepapers]
* [http://standards.ieee.org/getieee802/802.3.html Full text of the IEEE 802.3 standard]
* [http://www.ieee802.org/3/ IEEE 802.3 Ethernet Working Group]
* [http://grouper.ieee.org/groups/802/3/ae/ IEEE P802.3ae 10 Gbit/s Ethernet Task Force]
* [http://www.ieee802.org/3/an/index.html IEEE P802.3an (10GBASE-T) Task Force]
* [http://standards.ieee.org/reading/ieee/updates/errata/802.3-2005_Cor2-2007.pdf Corrigendum 2: IEEE Std 802.an-2006 10GBASE-T Correction]
* [http://grouper.ieee.org/groups/802/3/aq/index.html IEEE P802.3aq (10GBASE-LRM) study group]
* [http://www.ethernetalliance.org Ethernet Alliance website]
* [http://www.iol.unh.edu/consortiums/10gec/ University of New Hampshire Interoperability Laboratory 10 Gigabit Ethernet Consortium]