草榴社区 Cloud
Cloud native EDA tools & pre-optimized hardware platforms
Request a Free Trial →
PCIe PHY IP Enabling U.2/U.3 Form Factors for Hyperscale Data Centers
Manu Verma, Staff Product Marketing Manager, 草榴社区
Deployment of Artificial Intelligence (AI) and edge compute is driving a paradigm shift in hyperscale data centers. Trends such as 5G for AI-powered IoT edge applications, large amounts of data for video streaming, and zettabytes of data for fully autonomous vehicles, have required hyperscale data centers to support exponential growth of data volume and implement distributed low-latency processing. Such trends have also led to more complex and expensive data center storage architectures, which are required to support a combination of hard drives and solid-state drives (SSDs) configured with varying interfaces for different workloads. This article explains how to use PCI Express (PCIe) 5.0/6.0 PHY IP to enable U.2/U.3 connectivity in data center storage designs.
Evolution of Storage Architectures
One major challenge that system companies face when designing servers is upgrading the storage architecture to meet present and future data center requirements. The upgrade includes integration of multiple backplanes, mid planes and controllers, all of which increase system complexity and cost.
One way to address this challenge is to optimize storage architecture by implementing drive consolidation that support all three SSD protocols - (Serial Attached SCSI (SAS), SATA and NVMe - with one common infrastructure.
Initially, SAS interface allowed SATA SSDs/HDDs (hard disc drives) to interoperate with SAS backplane, Host Bus Adapters (HBAs) or RAID Redundant Array of Independentdent Disks (RAID) controllers. However, support was unavailable for NVMe SSDs, which required a separate configuration that utilized an NVMe-enabled backplane.
Figure 1 shows a server storage architecture with the ackplane, expander or mid-plane and controller.
Trending Now
Figure 1: Storage architecture requiring different backplanes for SAS, SATA and NVMe
The advent of the U.2 specification based on SFF-8639 form factor was the first step towards storage drive consolidation. The U.2 form factor supports up to four lanes of PCI Express (PCIe) for NVMe SSDs, and up to two lanes for SAS/SATA SSDs/HDDs, as shown in Figure 2. Even though U.2 supports all three drive interfaces, NVMe, SAS and SATA, it is not optimized since it does not provide interchangeable SAS/SATA/NVMe support in the same slot. It still requires a separate backplane, mid-plane and controller for NVMe.
Figure 2: The U.2 form factor supports up to 4 lanes of PCIe for NVMe SSDs and up to 2 lanes of SAS/SATA SSDs/HDDs
The storage architecture further evolved with the advent of the U.3 specification, which is built on the SFF-8639 connector. U.3 provides true storage drive consolidation by supporting all three drive interfaces in the same server slot using 1 backplane, 1 mid-plane and 1 controller as shown in Figure 3. U.3 was developed by the Storage Networking Industry Association (SNIA) SSD Form Factor (SFF) Technical Affiliate (TA) and defined by the SFF-TA-1001 specification. It is also backward compatible with U.2, but U.2 is not compatible with U.3 hosts.
Figure 3: The U.3 specification supports 1 backplane, 1 mid-plane and 1 controller to provide true interface drive consolidation
The U.3 specification primarily includes a tri-mode controller, SFF-8639 connector and a Universal Backplane Management.
- Tri-mode controller: Forms an interface between server host and universal backplane. It supports respective protocols for SAS, SATA and NVMe drives, providing server system companies a consolidated solution instead of using separate controllers for SAS/SATA and NVMe. This provides flexibility and allows blending the drives as per the end application requirements.
- SFF-8639 connector: This connector is placed on the drive bays and connects SAS, SATA and NVMe drives to the tri-mode controller. The host detects a compatible protocol for communication. The SFF-8639 connector uses a combination of available pins to identify which drive is present. U.3’s storage drive consolidation capability enables server system companies to simplify their backplane designs by using a single connector to support SAS/SATA and NVMe drives, as shown in Figure 4. This also results in cost benefits due to the utilization of fewer components.
Figure 4: U.3 connector supports SAS, SATA and NVMe drives
- Universal Backplane Management (UBM): Developed by the SSD Form Factor Working Group and defined by the SFF-TA-1005 specification, UBM provides a standard method for managing and controlling SAS, SATA and NVMe backplanes for configurability and flexibility. UBM enables two important aspects of backplane management:
- Support interchangeability between storage media types within a drive bay, for instance changing between SAS, SATA and NVMe while the system is running
- Ability to manage SAS, SATA and NVMe devices without any changes to storage drives or software stacks
- Support interchangeability between storage media types within a drive bay, for instance changing between SAS, SATA and NVMe while the system is running
Advantages of U.3 Form Factor
The U.3 specification
- Provides higher flexibility and configurability with reduced system complexity and cost by consolidating SAS, SATA and NVMe drives and simplifying the system with cost-effective cables and components
- Enables implementation of SAS, SATA and NVMe protocols through a single drive slot
- Enables standard management tools across server storage protocols
- Consolidates the benefits of SAS, SATA and NVMe drives for better performance
- Offers the option to support NVMe, which is gaining traction over SAS and SATA
- Maintains backward compatibility with U.2
PCIe PHY IP Support for U.2/U.3 Form Factors
The enterprise storage is migrating from SATA to NVMe due to benefits like scalability and lowest latency. Hyperscale data centers require operational data for faster storage such as NVMe SSDs with PCIe PHY interface, which is supported by U.2/U.3 architectures. PCIe 4.0 PHY IP is already deployed in NVMe SSDs and PCIe 5.0 is becoming the prominent interface as the performance requirement is doubling from 16GT/s to 32GT/s.
As shown in Figure 5, a customized, area-optimized, 6-lane PCIe 5.0 PHY solution can be used to support U.2/U.3 connectivity with features that include:
- Reference clock detection circuit with hysteresis and a digital clock edge counter
- Support for 25MHz reference clock
- Spread spectrum clocking (SSC) and Separate Reference clock Independent SSC (SRIS) for 25MHz reference clock
- A unified Physical Coding Sublayer (PCS) wrapper with bifurcation multiplexer (MUX) and any lane to any MUX
The leading storage companies are already using such solution.
Figure 4: Customized PCIe 5.0 PHY for U.2/U.3 connectivity
SSD companies are already designing PCIe 6.0 PHY + Compute Express Link (CXL) IP solutions for U.2/U.3 connectivity. PCIe 6.0 PHY IP will improve memory bandwidth and CXL IP will lead to development of new storage architectures as it will enable storage systems to take advantage of much larger memory pools for caching.
Summary
With the exponential increase in data volume and complexity, hyperscale data centers are going through a paradigm shift, incorporating distributed low latency processing. Implementation of U.2/U.3 storage architecture is critical for these data centers as it enables varying workloads with optimized performance and cost. The U.3 standard ensures that the storage is designed to meet present and future data center requirements. The specification supports SAS, SATA and NVMe drives in the same server slot using 1 backplane, 1 mid-plane and 1 controller. In addition to system flexibility, the U.3 specification also provides a replacement path between SAS, SATA and NVMe. U.3 results in simplified backplane system and reduction in total cost of ownership due to fewer traces, cables and connectors. It also maintains backward compatibility with the U.2 platform. To easily enable U.2/U.3 connectivity, designers can integrate 草榴社区’ DesignWare? IP for PCIe 5.0, PCIe 6.0 and CXL. The available 草榴社区 DesignWare IP for PCIe 5.0 offers a customized x6-lane solution, enabling U.2/U.3 formfactors. Designers can leverage 草榴社区’ DesignWare IP for PCIe 6.0 to address future requirements for U.2/U.3. 草榴社区 offers a complete PCIe controller, PHY and verification IP solution that is silicon proven and has achieved successful interoperability with third-party devices. The solution is optimized to meet latency, area, and power requirements of the target application.