草榴社区

An Introduction to the CXL Device Types

VIP Expert

Mar 03, 2020 / 4 min read

Compute Express Link (CXL) is the latest specification in interconnect technology for high bandwidth devices. It provides high-speed, efficient connectivity from CPUs to other components of the high-performance computing platform.

CXL’s coherent memory access capability between a host CPU and a device, such as hardware accelerators, meets the requirements for handling data and compute intensive workloads in next-generation designs by leveraging the advanced capabilities of PCIe architecture.

Let’s explore the types of CXL devices and the verification challenges that are unique to CXL, like maintaining the cache coherency between a host CPU and an accelerator. For an initial overview of this specification, check out our previous blog here.

Type 1 CXL Device

  • Implements a fully coherent cache but no host-managed device memory
  • Extends PCIe protocol capability (for example, Atomic operation)
  • May need to implement a custom ordering model
  • Applicable transaction types: D2H coherent and H2D snoop transactions

Type 2 CXL Device

  • Implements an optional coherent cache and host-managed device memory
  • Typical applications are devices which have high-bandwidth memories attached
  • Applicable transaction types: All CXL.cache/mem transactions

Type 3 CXL Device

  • Only has host-managed device memory
  • Typical application is a memory expander for the host
  • Applicable transaction types: CXL.mem MemRd and MemWr transactions

A CXL.cache/mem design has to maintain cache coherency between the host and device agent caches and their corresponding memory. The Bias Based Coherency model for a Type 1 CXL device defines two states for device attached memory – the host bias and the device bias. Each possess their own verification challenges.

CXL acts as a high-performing I/O interconnect system, trying to ensure reliable and efficient access to memory distributed throughout various components. Some of these components optimize performance by making use of the local cache and reduce the overhead of memory access. To support this type of configuration, CXL.cache protocol ensures that data held across the components in either memory or local cache remain coherent and consistent to each component. Device components in CXL are typically used as accelerators for computationally intensive applications, and hence contain a local cache. So, if the host component wants to access the same location of memory, it can then force the device to evict the line from its local cache and update the corresponding memory location depending on the cache line state.

The CXL.cache protocol defines interaction between the device and host as a number of requests that each have at least one associated response message and sometimes a data transfer. The interface consists of three channels in each direction: Request, Response, and Data. The channels are named for their direction – D2H (Device to Host) and H2D (Host to Device).

CXL cache snoop transaction flow chart

A CXL based system uses the Bias Based Coherency Model to improve memory access performance. When the device-attached memory is in a host bias state, the device accesses it like a regular host-attached memory. If the device needs to access the device-attached memory it first sends a request to the host which will resolve coherency for the requested line.

Alternatively, when the device-attached memory is in device bias state, the device is guaranteed that the host does not have the line cached. In this case, the device can access it without sending any transaction to the host.; however, the host sees a uniform view of device-attached memory regardless of the bias state.
In both the host bias state and the device bias state, coherency is always preserved for device-attached memory.

CXL.cache H2D Snp Transaction exchange:

  1. Host sends H2D request SnpData to the device
  2. Device receives the request and sends D2H response as RSPI_FWDM to the host.
  3. Device sends D2H data to host
  4. After host receives the response transaction is complete
CXL device connection diagram

A Type 3 device is primarily a memory expander for the host memory. The device operates over CXL.mem to service requests sent from the host. Two types of flows are supported for Type 3 devices – Read and Write flows.

CXL device configuration diagram
CXL device integration diagram

草榴社区 is the market leader for Design IP and Verification IP for CXL, contributing significantly to the evolution of the CXL ecosystem. Stay tuned for more deep dives on this new CXL specification, including topics like additional transaction types, layered architecture and the verification requirement/challenges of CXL designs.

For more information, please visit http://synopsys.com/vip

Continue Reading