草榴社区

How to Maximize PCIe 6.0’s Advantages with End-to-End PCIe Design 草榴社区

Gary Ruggles, Madhumita Sanyal

Jun 16, 2023 / 9 min read

If you dream of networking at the speed of the latest 800G Ethernet, you’ll need a serial bus interface to match it. Data center solid state drives (SSDs) and the accelerators used in AI applications are also hungry for speed. That’s why PCI Express? (PCIe?) 6.0 is gaining traction in AI, HPC, and data centers. PCIe 6.0 operates at 64GT/s and that’s lightning fast, twice as fast as the previous generation. But while network servers, SSDs, switches, and AI accelerators are all early adopters of PCIe 6.0, and network interface cards (NIC) and CPU host chips are on the horizon, how do you know if PCIe 6.0 is right for your next chip design?

Every time you advance, everything gets a little bit tougher, even as the time-to-market (TTM) pressure increases. If you want to take a step forward into a first mover position, but are hesitating, here’s a tip: Leveraging the leader in complete end-to-end PCIe hardware design tools can ease your journey to achieving industry-leading quality and security while lowering your risk. If you are considering making the move from PCIe 5.0 to 6.0, this overview will cover what you need to consider before getting into the PCIe 6.0 fast lane.

Hyperscale Data Center

What’s the Big Deal about PCIe 6.0 Anyway?

PCIe 6.0 is a transformative serial bus interface technology, a kind of sea change in interconnect based on several technological shifts in this version of the specification:

  • PAM-4 electrical signaling modulation scheme: Pulse amplitude signaling at four voltage levels produces three eyes, a shift from traditional non-return to zero (NRZ) signaling. Precoding and forward error correction (FEC) will reduce errors for analog and digital, respectively. This delivers 64GT/s bandwidth with low latency.
  • Flow control unit (FLIT) packet delivery: This new architecture for packet delivery (required due to the FEC) not only supports the increased bandwidth but also enables your system to support it.
  • L0p low-power state: The new L0p low-power state allows some lanes to go into a sleep mode as bandwidth requirements decrease in the system. This gives you the ability to optimize your power consumption while never shutting down the link.
  • Data integrity and security protections: The specification uses data object exchange (DOE) as the PCIe security building block at lower bandwidth levels with cryptographic data and keys. Component Measurement Authentication (CMA) provides a firmware cryptographic signature. Integrity and Data Encryption (IDE) gives your system packet-level security to prevent physical attacks. By coupling IDE with your controllers, it’s efficient security at high-bandwidth 64GT/s speeds.

The Challenges of a New Generation of PCI Express

If your application has the need for the speed and bandwidth that PCIe 6.0 has to offer, there are a couple important factors to consider in your adoption:

  1. The expense incurred in adopting a new PCIe specification: Added costs can come from moving to a smaller foundry process, new IP, and what the implication of adoption means to the whole system, including losses associated with cables and backplanes and adding new cards.
  2. The timing of PCIe 6.0 maturity: The PCIe 6.0 ecosystem hasn’t solidified and compliance, gated on  consortium workshops, is roughly a couple years out.

Any time you advance to a new generation of a standard there are cost considerations. Whether these costs are born now or later depends on your urgency in realizing the advantages PCIe 6.0 has to offer weighed against your goals and unique market pressures.

In terms of PCIe 6.0 maturity, the current adoption trajectory is similar to previous generations of the spec. Because PCIe is ubiquitous, you can be assured that the ecosystem to support the 6.0 generation is growing. You could wait until completion of consortium workshops, for the proverbial dust to settle, and for the spec to have a track record of product in the field before adopting it. But, adopting PCIe 6.0 now gives you all the amazing advantages it has to offer while helping to future proof your design, enabling you to be on the leading edge of innovation as the 6.0 ecosystem comes online.

Backwards Compatibility Hedges Your Risk

While engineers love designing advanced systems and technologies, they are also notoriously risk averse, continuously walking the fine edge of decisions that can determine first movers, fast followers, the muddle in the middle, late to market, or dead on arrival. Despite the significant technology advancements, PCIe has carefully written the specification for backward compatibility with previous versions of the spec. In other words, if you adopt the spec early, you don’t have to rely on a robust PCIe 6.0 ecosystem for your system to work. Whether it’s an endpoint or a complex system, PCIe 6.0 designs can plug into technology that uses any version of the PCIe standard, including PCIe 1.0 @ 2.5GT/s. If a device cannot support the new PCIe 6.0 64GT/s data rate, the link will be negotiated at the highest data rate supported by both link partners.

One note on backward compatibility, once a link negotiates 64GT/s FLIT mode, it must remain in FLIT mode: if an anomalous event impacts signal integrity—for instance, if a giant motor turns on causing a glitch in the power supply, or there is extra noise, or somebody moves a cable—the system may need to fall back to the previous PCIe generation data rate (such as 32GT/s or even 16GT/s). Even when a glitch such as this causes a fall back, the system will remain in FLIT mode. Supporting a previous generation while remaining in FLIT mode is a new state that never existed before. While this is not necessarily likely, it is a possible complication. To hedge for this possibility, you must support FLIT mode for all PCIe data rates.

Backward compatibility is one of the key beauties of the spec. PCIe 6.0 is already part of a whole PCIe ecosystem that dominates the industry, providing ubiquitous connectivity. Afterall, you cannot change the entire data center and all the devices in it at once when you move from Gen. 5 to 6. PCIe enables a fall back option, so the whole universe doesn’t need to upgrade simultaneously.

Network Switch Design with PCIe 6.0

Navigating network switch design development illustrates the complexity of an implementation of PCIe 6.0 in practical application. To get the advantages of 64GT/s and PAM-4 signal integrity in a design with 256 lanes, for instance, you will not only need the full switching solution, but you will also need to integrate with technology beyond your system. You need to think through the co-design of the package, the board, and all the integrations, as well as how you put it together, into a holistic solution. This entails activities such as optimizing the bump map, reduction of escaping signals, and optimizing the beachfront of your die, on all die edges. On top of activities like these, and more, you must make sure that you get the lowest cost package for the device.

Designing a switch with PCIe 6.0 requires an understanding of a vast array of different perspectives so the system will work holistically. Implementing PCIe 6.0 in a switch requires different configurations with multiple links and multiple controllers. Because of this, you will need an IP provider who has deep experience in PCIe and has already done the analysis for PCIe 6.0, including working with customers on PCIe 6.0 switch designs.

To achieve PCIe 6.0 success in switches or any other application, you need a partner who can provide you with expertise and a complete solution that maps to the end-to-end design process, from IP and design through production. A solid design partner can help you lower risk, overcome the challenges of complexity, and ease your journey to success.

Is PCIe Verification a Big Deal Again?

Yes.?The changes introduced with PCIe 6.0 affect all layers, creating increased verification complexity.

At the physical layer, 64GT/s speed support is achieved using PAM4 encoding. PCIe 6.0 introduces 256B FLIT, which demands certain packing rules for protocol packets into FLITs, increasing design complexity. It also brings in FEC complexity along with the existing CRC mechanism.

For backward compatibility reasons, 256B FLITs are supported at 2.5/5/8/16/32 GT/s speeds. This demands verification of FLIT mode at all supported speeds.??

At the data link layer, addition of new DLLP types–namely optimized updatefc and link management, for exchanging link information, and change in sequence number/replay rules–demands in-depth verification of sequence numbering, FLIT replay command handshakes, and selective/full replay mechanisms to provide guaranteed FLIT transfer across to the link partners. 

Along with the introduction of FLIT, new TLP framing rules are also defined, which requires extensive verification.

PCIe 6.0 also introduces a new power state L0p, which enables power reduction without impacting the traffic flow. Link management DLLPs are used to establish L0p handshake between link partners. This adds to the design complexity necessitating in-depth verification.

The disruptive nature of PCIe Gen6 specification will create new verification challenges not only for backward compatibility, bandwidth, and performance of the interface but also for dependent NVMe, SSD, and other PCIe-based storage technologies. 草榴社区 Verification IP (VIP) and test suite are designed to handle this verification complexity. 草榴社区 VIP is used to verify silicon-proven 草榴社区 IP.

In addition, running system-level payload on SoCs requires a faster hardware-based pre-silicon solution. 草榴社区 transactors based on?草榴社区 IP?enable fast verification hardware solutions including?草榴社区 ZeBu??emulation systems and?草榴社区 HAPS??prototyping systems for validation use cases. 

Why is a Complete PCI Express 6.0 Solution Important?

Because 草榴社区 engages with a lot of customers, we have a large knowledge base and expertise to build regression algorithms so you can simulate your performance from the get-go. We use our experience with a wide array of different customer configurations to do this: from complex controller configurations to a variety of lane combinations comprised of many links. We also have our subsystem team who does different physical implementation studies with many different controllers and configurations.

When we work with you, it’s not solely about giving you the PHY, the controller, the IP, the IDE security module, and the verification IP. We also give you guidance in terms of your connectivity, how to bring up your simulation, how to do your back-end synthesis (including whether it must be hierarchical or flat), advising on the physical placement (the spacing requirement and where you put the blocks), clock rebuilding, balancing, routing, timing, and closure guidelines, and the timing-critical path—all of it. We lead the industry in supporting a wide range of features and capabilities to debug firmware and hardware and optimize power, performance, and area (PPA), and latency. And, 草榴社区 is the early leader in providing a proven IDE module, so you can implement it in your hardware and ensure your design is secure.

PCIe controller

As the earliest in the industry to offer complete solutions for the next generation, 草榴社区 occupies an advantageous spot in the PCIe 6.0 game. We have a long history of working closely with our customers through difficult PCIe issues—when specs aren’t finalized or even when IP isn’t finalized. We work hand-in-hand through very complicated scenarios to help you successfully introduce your chip early in your industry, and we are a critical player in first-time, full tape-out silicon success. 草榴社区 continues to lead the PCIe path with our IP performing at 128 GT/s at this year’s PCI-SIG Developer Conference. We also showcased 草榴社区’ extensive PCIe 6.0 interoperability with seven demos with our partners, including two end-to-end host-to-device system demonstrations.

As the earliest in the industry to offer complete solutions for the next generation, 草榴社区 occupies an advantageous spot in the PCIe 6.0 game. We have a long history of working closely with our customers through difficult PCIe issues—when specs aren’t finalized or even when IP isn’t finalized. We work hand-in-hand through very complicated scenarios to help you successfully introduce your chip early in your industry, and we are a critical player in first-time, full tape-out silicon success. 草榴社区 continues to lead the PCIe path with our IP performing at 128 GT/s at this year’s PCI-SIG Developer Conference. We also showcased  草榴社区’ extensive PCIe 6.0 interoperability with seven demos with our partners, including two end-to-end host-to-device system demonstrations.

Watch some conference highlights in this video:

Our 20-year relationship with the PCI-SIG consortium includes a seated board member, and deep insights on the specification. While there is no one-size-fits-all answer to design for PCIe 6.0, there is the 草榴社区 one-stop shop that offers market-leading PCIe 6.0 design solutions that are stable, vetted, interoperated, and the most complete solutions available on the market today. If you are interested in learning more about our 草榴社区 PCIe 6.0 solutions, contact us.

Continue Reading