²ÝÁñÉçÇø

SoC performance continues to see generational upgrades and there has been much discussion on the importance of multi-core designs to drive performance upgrades as Denard Scaling has been argued to be ending (Figure 1).

Regardless of this phenomenon, to satisfy performance needs, foundries continue to drive next-generation process nodes that enable higher frequencies and higher logic densities to incorporate more processing elements, all at reduced power footprints.  This continued innovation is shown in Figure 2.

Multi-core architectures and processor element arrays, in addition to process node innovation, continue to aggressively solve generational performance gains. However, it¡¯s not only multi-core innovations, nor process node generational upgrades that require a new multi-die system architecture.  The memory wall is clearly one of the key drivers for multi-die systems (Figure 3).  The chart shows that memory densities are progressing at 2x every 2 years vs a workload that is increasing by 240x every 2 years in terms of memory densities. For more information, see the blog post on Medium.com entitled .

AI and security workloads have increased dramatically, and this has driven many on-chip performance innovations beyond frequency increases.  Many of these innovations have focused on the processors required for these workloads. AI algorithms have driven design of massive multiply accumulate parallelism as well as creative nested loops to reduce cycles and increase work completed per cycle. These workloads however also require much larger memory densities to store the weights, coefficients, and training data.  This has driven larger capacity and higher bandwidth memories both on-chip and off-chip.  For off-chip memories, the indusry has adopted next-generation HBM, DDR, and LPDDR rapidly. However, it is the on-chip memory configurations that are providing vendor differentiations.  Examples of this include the AI Accelerator space where each vendor is integrating more memory density at higher bandwidths in terms of global SRAMs and caches.  Unique methods to optimize the memory configurations per processing element are also a critical piece of the innovation puzzle.  

Taking a step back from the SoC-specific performance gains, a critical industry problem is power usage of these AI systems within the cloud.  Figure 4 shows Google Data Center¡¯s power usage.  Clearly the design activity to build an SoC with a more efficient CPU is critical and multiple SoC startups openly promote their AI processor efficiency to solve this problem.  But the overall system performance also includes the off-chip memory as seen in the DRAM power consuming 18% of the total.  Designing for lower power has seen an increase in HBM adoption because the pJ/bit performance is unmatched by other technologies.

We¡¯ve discussed the adoption of off-chip memory interfaces such as DDR/LPDDR and HBM to improve memory bandwidth however these technologies are not keeping up with the ability to integrate processors for AI workloads within the die.  The off-chip memory trajectory gap is clearly increasing. Meta pointed to this trend in an OCP summit presentation in recent years (Figure 5).

Multi-die systems are one of the hottest topics in the industry. However, the technical challenges related to memories to satisfy the current AI and security workloads are clearly driving next-generation SoC architecture innovations.  These architectures need higher performance, and more memory per processing element as technology process nodes advance.  If the memories are scaling at smaller rates than the processing elements but the workloads are demanding more memory per processing element, there must be technology disruptions.  One clear solution has been multi-die systems, leveraging more on-chip memories at higher bandwidths and improved densities.  These memory and IO innovations will see rapid adoption to meet the needs of future workloads and may open opportunities for future industry disruptions.