Cloud native EDA tools & pre-optimized hardware platforms
Priyank Shukla, Staff Product Marketing Manager, 草榴社区
Hyperscale data centers need to scale appropriately as increased demand is added to the system. For the required server-to-server communication in hyperscale data centers, Ethernet has become the primary network protocol of choice as it allows hyperscalers to disaggregate network switches and install their software operating systems independently. Ethernet enables cost-effective, dense, open switches and networking technologies which reduce cost/power per bit with transistor scaling. Ethernet is a computer networking technology that defines physical and data-link layers of the Open Systems Interconnection (OSI) model. The IEEE 802.3 standard describes these functions in an architectural way with emphasis on the logical division of the system and how they fit together. Data link layer, consisting of Media Access Control (MAC) creates Ethernet data frames and uses the underlying Ethernet physical layer to transfer the data frame through a medium.
This article describes the Ethernet PHY used in high-performance computing (HPC) system-on-chips (SoCs) and how an integrated MAC + PHY IP can accelerate path to compliance and design closure.
Ethernet physical layer or PHY, as an abstraction layer, transmits and receives data. The PHY encodes data frames for transmission and decodes received frames with a specific modulation speed of operation, transmission media type and supported link length.
The 草榴社区 article, Understanding the Ethernet Nomenclature – Data Rates, Interconnect Mediums and Physical Layer, describes in more detail the Ethernet PHY nomenclature, which is based on data rate, modulation, and media type. Speed-specific Media Independent Interfaces (MIIs) allow use of various PHY devices for operation over different media types such as twin axial copper (BASE-C), twisted pair (BASE-T), electrical backplanes (BASE-K), or fiber optic cables (BASE-L/R).
For example, most personal computer users are familiar with “Ethernet cables” in their laptops/PCs. Figure 1 shows a simplified block diagram describing how data is transferred to and from the processor over an Ethernet cable. In this use case, Ethernet data frames (packets of data), assembled by Ethernet MAC in the CPU, travel across the mother board (a printed circuit board) through MII/GMII, defined by the IEEE802.3 standard, before reaching an Ethernet PHY which transmits electrical signals over twisted pair cables through RJ 45 connectors.
Figure 1: A simplified example of Ethernet data packets traveling from the processor to the Ethernet PHY in a personal computer
Figure 2 shows a data center as a network of compute and storage systems connected with optical and copper media. Optics offers a power-efficient way for long-distance Ethernet links with Single Mode Fiber (SMF), providing the longest reach as it uses single path through the fiber. Multi-Mode Fiber (MMF) offers a more cost-effective alternative to SMF and is typically for disctances of 500 meters or less. A server rack unit to Top of the Rack (TOR) switch links are generally implemented using twin-axial copper cables, generally referred to as Direct Attach Copper (DAC) cables.
Figure 2: Usage of different types of optics in a server-to-server communication in a data center
Figure 3: Movement of data packets in a rack unit of a server farm
Figure 3 shows data packets traveling in a data center, originating from a processor in one of the rack units of a server farm. Data from the processor goes to the network interface card (NIC) through a PCIe interface. The network interface card creates Ethernet frames by implementing MAC functions. The frames travel to the Top-of-Rack (ToR) switch through a twin-axial copper PHY or DAC cable. Depending on the DAC cable length and the switch silicon in ToR rack unit physical location, retimers may be used. These retimers implement back-to-back backplane Ethernet PHYs to extend the reach of electrical signals. The ToR switch routes the frames and the optical module converts the medium from electrical to optical by implementing both electrical and optical PHY functions.
IEEE802.3-2018 and Ethernet Technology Consortium (ETC) have defined the 400 Gb/s and 800 Gb/s standards respectively. It is important to note that the 800 Gb/s Ethernet is based on 400 Gb/s Ethernet Access Method and Physical Layer standards from IEEE 802.3-2018 and 802.3ck.
Figure 4: 400 Gb/s Ethernet PHY architecture
Figure 4 illustrates the 400 Gb/s Ethernet PHY from an architectural abstraction level, showing that an 800 Gb/s or 400 Gb/s electrical Ethernet PHY implements:
An integrated Ethernet PHY IP that includes the PCS, PMA, PMD and auto negotiation functionality enables faster adoption of the latest 800Gb/s and 400Gb/s Ethernet. Figure 5 shows an example implementation of an 800G/400G PCS.
Figure 5: Ethernet PCS block diagram
Ethernet has become the defacto standard for server-to-server communication in modern HPC data centers. Ethernet data frames travel through the server units over various channels and media types. Integrating the MAC and PHY in an Ethernet system reduces design turnaround time and offers differentiated performance. 草榴社区 provides a complete 200G/400G and 800G Ethernet controller and PHY IP solution that includes the PCS, PMD, PMA and auto negotiation functionalities, as shown in Figure 6.
Figure 6: Complete 200G/400G and 800G Ethernet controller and PHY IP solution
The DesignWare? 112G Ethernet PHY IP delivers exceptional signal integrity and jitter performance which exceeds the IEEE 802.3ck and OIF standards electrical specifications. The area-efficient PHY demonstrates zero BER with more than 42dB channel loss and offers power efficiency of less than 5pJ/bit. The DesignWare 200G/400G and 800G Ethernet MAC and PCS support IEEE 802.3 and consortium specifications including Reed Solomon Forward Error Correction (FEC) and timestamping with low jitter for maximum precision.